Most people learning web automation start in the wrong place. They pick a tutorial, install a browser driver, and immediately try to build the thing they actually want — a bot that monitors twelve sites, logs into three of them, and emails a dashboard. Two evenings later they are debugging a race condition inside a scheduler inside a headless browser, and they conclude they are "not technical enough."
They were fine. The project was just three projects wearing a trench coat.
The key takeaway: build in this order — a form fill, a scheduled check, then a data pull. Each one adds exactly one new hard skill, and each one is finishable in an evening or two. And build the ethics in from the first script, not as a retrofit, because the habits you form on project one are the habits you will still have when someone is paying you.
Lesson one, before any code: what you are allowed to touch
This is not a disclaimer section. It is a practical skill, and it is the one that separates people who get hired for automation work from people who get their IP blocked.
Three rules, from the first line of code:
- Automate things you own or have permission to use. Your own site, your own staging environment, a client's system you were hired to test, or a public source that publishes terms permitting it. "It loaded in my browser" is not permission.
- Read
robots.txtand the terms of service before you write the loop, not after. They tell you which paths are off-limits and often state a crawl delay. Honouring them is free; ignoring them is how a portfolio project becomes a legal problem. - Rate-limit yourself lower than you think you need to. One request every two to five seconds is fine for almost any learning project. Your script's speed is worth far less than the access you lose by being rude.
One hard line, stated plainly: automation for testing, monitoring, accessibility checks and lawful data collection is legitimate work with a real job market behind it. Automation aimed at defeating someone's authentication, creating accounts en masse, or spamming is not a career, it is a liability. Everything here assumes the first kind.
Project one: fill a form (on a site you own)
Stand up a simple form — a contact form on your own site, a demo app, or a local page you wrote — and write a script that opens it, fills the fields, submits, and asserts that the success message appeared.
That sounds trivial. It is not, and the reasons it is not are the entire lesson:
- Selectors. You learn why
#emailsurvives a redesign andbody > div:nth-child(3) > form > input:nth-child(2)does not. This is the single most transferable idea in the whole field. - Waiting properly. Your first version will fail intermittently because the script typed before the field existed. Fixing it with
sleep(3)works and is wrong; fixing it with an explicit wait for "element is visible and enabled" works and is right. Understanding that difference is the moment you stop being a beginner. - Assertions. An automation that does not check its own result is not automation, it is a rumour. Every script should end by verifying something specific.
Ship it when it can run ten times in a row without a manual fix. That reliability bar — not the feature list — is what makes automation valuable.
Project two: a scheduled check that tells you something changed
Now take something you genuinely want to know — is my site up, has this page's price changed, did the status page go yellow — and run a check on a schedule.
New skills this forces:
- Scheduling.
cronon Linux/macOS, Task Scheduler on Windows, or a hosted scheduler. You will immediately discover that a script which works in your terminal fails under cron because of paths and environment variables. Every automation engineer learns this exactly once. - State. "Tell me when it changes" means the script must remember last time's value. A tiny JSON or SQLite file is enough, and now you are thinking about persistence.
- Idempotence and quiet failure. What happens if the check runs twice? If the site is down, do you alert once or every five minutes forever? Real monitoring answers these deliberately.
- Notification. Email, a webhook, a chat message. Cheap to add, and it converts a script into something a non-technical person can benefit from — which is the beginning of it being worth money.
Here is the shape of a polite, stateful check in Python:
import json, time, pathlib, requests
STATE = pathlib.Path("state.json")
URL = "https://example.com/status" # a page you own or are permitted to poll
HEADERS = {"User-Agent": "my-monitor/1.0 (contact: [email protected])"}
def load_state():
return json.loads(STATE.read_text()) if STATE.exists() else {}
def check():
r = requests.get(URL, headers=HEADERS, timeout=20)
r.raise_for_status()
return r.text.strip()[:200]
state = load_state()
current = check()
if state.get("last") != current:
print("CHANGED:", current) # replace with your notification
STATE.write_text(json.dumps({"last": current, "seen": time.time()}))
else:
print("no change")
Two details worth copying: the honest User-Agent with a contact address — site owners block anonymous bots far faster than identified ones — and the timeout, without which a script will one day hang forever and take your scheduler with it.
Project three: pull structured data you are allowed to have
The third project is a small collector: walk a set of pages you have permission to read, extract a few fields, and store them in CSV or SQLite so you can query them later.
This is where the remaining fundamentals arrive:
- Parsing. HTML into structured fields, and the discovery that a documented API — if one exists — beats parsing every time. Always check for one first.
- Pagination and stopping conditions. Knowing when you are done, and never letting a loop run unbounded.
- Retries with backoff. Networks fail. Retry twice with increasing delay, then give up loudly. Infinite retry is how you accidentally attack someone.
- Politeness as code. A fixed delay between requests, a concurrency cap, and an early exit on repeated
429 Too Many Requestsresponses. - Schema thinking. Deciding what a row means before you collect ten thousand of them.
If you want the wider context on turning a technical skill like this into income, our guide to learning skills that pay covers how to choose one and how long it realistically takes to get to employable.
The wall everyone hits: challenges and CAPTCHAs
Somewhere in project two or three you will hit a page that stops your script with a CAPTCHA or a browser challenge — often on your own QA environment, because the same anti-abuse layer that guards production is usually enabled in staging too.
The correct first move is always the sanctioned path: a test-mode key, an allowlisted IP for your CI runner, an API endpoint, or a staging flag your team can disable. Ask before you engineer around anything.
When the target is genuinely yours and no such switch exists — a regression suite that must run nightly against a protected form, say — a solving service is the pragmatic tool. CaptchaAI is one, and a useful one to look at while learning, because its interface teaches a pattern you will meet everywhere: submit a task, receive an ID, poll for the result.
It uses the legacy 2Captcha-shaped protocol — POST to /in.php on ocr.captchaai.com to submit, then poll /res.php with action=get&id=<taskId> — with a 32-character API key and json=1 for structured responses (omit it and you get plain text). Documented poll cadence is about five seconds until CAPCHA_NOT_READY stops coming back:
import requests, time
API = "https://ocr.captchaai.com"
KEY = "YOUR_API_KEY" # 32-character key
task = requests.post(f"{API}/in.php", data={
"key": KEY, "method": "userrecaptcha",
"googlekey": "SITE_KEY", "pageurl": "https://staging.example.com/form",
"json": 1,
}, timeout=30).json()
task_id = task["request"]
while True:
time.sleep(5) # documented ~5s cadence
res = requests.get(f"{API}/res.php", params={
"key": KEY, "action": "get", "id": task_id, "json": 1
}, timeout=30).json()
if res["request"] == "CAPCHA_NOT_READY":
continue
if res["request"] in ("ERROR_UNSOLVABLE", "ERROR_ZERO_BALANCE"):
raise RuntimeError(res["request"])
token = res["request"]
break
The polling loop, the sentinel "not ready" value, the named error codes you must branch on — that is the async-job pattern, and once you recognise it you will see it in payment APIs, video encoders and report generators too. Learning it here is not wasted.
Its published per-type figures (the vendor's own numbers) include image CAPTCHAs at over 99% success in under half a second across 27,500+ variants, reCAPTCHA v2 at over 99.5% in under 60 seconds, and Cloudflare Turnstile at 100% in under 10 seconds. Pricing is thread-based rather than per-solve: you buy concurrent threads — BASIC is $15/month for 5 threads, ADVANCE $90/month for 50 — with unlimited solves per thread and no daily caps. For a learner that matters mainly because a runaway loop cannot generate a surprise bill; it just queues.
None of this changes rule one. A solving service is for challenges standing in front of your own systems or lawful, permitted collection. It is not a tool for getting past someone's front door.
How to tell you have actually learned it
Four honest checks, and none of them is "finished a course":
- Can your script run unattended for a week without you touching it? Reliability is the product.
- Does it fail loudly and specifically? "Selector
#submitnot found on /contact after 10s" beats a stack trace every time. - Could someone else run it from your README? Environment variables documented, no hard-coded paths.
- Would you be comfortable showing the target site's owner exactly what it does? If not, you built the wrong thing.
Clear all four on the three projects above and you have a portfolio. That is genuinely enough to start taking small paid work — QA scripts, monitoring setups, report automation for a small business — which our earn online guide covers in more depth.
FAQ
Which language and library should I start with? Python with Playwright or Selenium, or JavaScript with Playwright. Pick the language you already know a little of; the concepts transfer completely, and arguing about the choice costs more time than learning either.
Do I need a browser at all? Often no. If the page returns the data in its HTML or through an API, plain HTTP requests are faster, cheaper and far more stable. Reach for a full browser only when the content is rendered by JavaScript or the flow genuinely needs clicks.
How long until this is employable? For small paid work, a few focused weeks is realistic if you finish all three projects to the reliability bar above. Treat anyone promising "job-ready in a weekend" as selling something.
Is web scraping legal?
It depends on the source, the data and your jurisdiction — which is exactly why the answer is to work within robots.txt, terms of service and rate limits, avoid personal data, and prefer official APIs. That posture is both the safe path and the professional one.
What if a site blocks me while I am learning?
Stop, slow down, identify your script honestly in the User-Agent, and switch to a target you own. Getting blocked is feedback that your request rate or your permission was wrong.
The short version
Three projects, in order. Fill a form, and learn selectors and waiting. Run a scheduled check, and learn state, scheduling and notification. Pull structured data, and learn parsing, backoff and politeness. Build the permission rules and the rate limits into project one so they are habits, not afterthoughts.
When a legitimate challenge blocks work on a system that is genuinely yours, CaptchaAI handles that layer with a simple submit-and-poll API and thread-based pricing, so you can keep the focus on the automation itself. And when you are weighing which skill to invest your evenings in next, browse the related everyday decisions at BeAdvices.
Small, finished, reliable beats ambitious and abandoned. Every time.
