Will this PySpark job survive the full data?
Paste or drop one PySpark job - the Python, its spark-submit script, spark-defaults.conf and
requirements.txt. Your browser follows every DataFrame from its read to its write and checks how
it will run: a collect() or toPandas() of data nothing bounds, a window with
no partitionBy, a join with no key, Python UDFs, count() for emptiness, actions and
withColumn inside Python loops, adaptive execution or broadcast switched off,
repartition(1) before a write, partitionBy on an id, inferred schemas, JDBC reads
through one connection, credential literals, and overwrites that delete every other partition. Free, before
you sign in, and nothing is uploaded. Then pick a lane: the audit gives exact edits for each
finding, or the fix lane writes the whole fixed job - and the page re-checks both.
The companies, buckets, hosts and values in the examples are invented. The first three are audits (the first also carries a saved fix run of its edited code, one click after the audit), the last a fix run; each comes with a saved model reply, so you can see the whole page without signing in or spending a credit.
The audit rates the driver, shuffles, compute and io, confirms or dismisses every flag below, adds what rules cannot see, and gives exact before/after edits for each finding - which the page applies to your paste and re-checks.
What this does, and what it does not
The check is a reading of the text, not Spark: nothing is imported or run. Each Python file is lexed so strings and comments are never mistaken for code, its blocks are rebuilt from indentation so every loop and function has its exact extent, every DataFrame is followed back through its assignments to its read, and each flag carries the catalogue id it comes from and the line it points at. Data sizes, key skew and the cluster are not in the code: whether a table is small enough to broadcast, or a key is hot, is a question you answer in the note, not something the page assumes. The Spark UI - stage durations, shuffle read, spill, and the slowest task against the median - remains the ground truth.
The paid lanes see your paste and the page's facts. Every quote and edit they return is checked against your paste, every edit and every returned file is put back and re-checked, and a reply that repeats a credential from your paste is called out. A paste stays in this browser unless you run a lane. Derived from the agent skill @wshobson/spark-optimization by Seth Hobson (wshobson/agents, MIT). Sources are listed in llms.txt.