# Skewcheck

> Audit one PySpark job before it runs at scale - will it kill the driver, run one job per loop pass, shuffle everything through one task, or delete yesterday's output - then fix it. The free check runs in the browser and uploads nothing: it lexes the pasted Python (plus the spark-submit script, spark-defaults.conf and requirements.txt), rebuilds its blocks from indentation, follows every DataFrame from its read to its write, inventories every read, write, action, join, window, UDF, cache and Spark setting, and runs 20 deterministic checks drawn from Spark tuning practice. The paid audit rates the driver, shuffles, compute and io, confirms or dismisses every flag and gives exact before/after edits per finding; the paid fix lane writes the whole fixed file(s) and a changes list tied to the flags. The page checks every quote and line against the paste, applies every edit and file back into the paste and re-runs its checks - including that every read and write is still there - and calls out a reply that repeats a credential from the paste.

App: https://skewcheck.skillsafe.ai/
API tutorial: https://skewcheck.skillsafe.ai/api.html

## Input

- The job pasted into one box, each file introduced by a `# file: <path>` line; dropped files or a dropped job folder are joined that way automatically. A paste with no `# file:` line is read as one file (job.py). requirements.txt or pyproject.toml gives the pyspark version (`isEmpty()` needs 3.3, adaptive execution is on by default from 3.2).
- Settings are read from the session builder (`.config`), `spark.conf.set`, a settings dict, spark-submit `--conf`/`--executor-memory`/`--executor-cores`/`--driver-memory`/`--num-executors`, and spark-defaults.conf.
- Nothing is imported or run: every flag is a reading of the text, with the file and line it points at. Strings and comments are lexed out first, so a string that mentions collect() is never a finding.
- Data sizes, key skew and the cluster are not in the code: SP-21 says so, and the note is where you give them ("dim_pages is 2 MB").

## Flags the free check raises

- SP-01 (high): collect() or toPandas() of a DataFrame whose lineage has no limit or aggregate (medium when the page cannot follow it back to a read).
- SP-02 (medium): a Python UDF (F.udf, @udf, spark.udf.register) or a df.rdd.map/flatMap/filter round trip; pandas_udf is listed, not flagged.
- SP-03 (medium): count() compared with 0 or used as a truth test, or len(collect()) == 0, to test emptiness.
- SP-04 (medium): Spark actions (count, collect, show, take, write ...) inside a Python loop or comprehension; one flag per loop with every line.
- SP-05 (medium): a DataFrame reassigned from itself with withColumn, union or join inside a Python loop.
- SP-06 (high): a Window spec with orderBy or a frame but no partitionBy.
- SP-07 (high): join with no condition, how="cross" or crossJoin (a spark.range or createDataFrame side is exempt).
- SP-08 (medium): spark.sql.adaptive.enabled, skewJoin.enabled or coalescePartitions.enabled set to false (low: pyspark before 3.2 with AQE never enabled).
- SP-09 (medium): spark.sql.autoBroadcastJoinThreshold set to -1.
- SP-10 (medium): repartition(1) or coalesce(1) before a write.
- SP-11 (medium): partitionBy on an id, key, session, timestamp or other high-cardinality column name.
- SP-12 (medium): inferSchema on a read (low: a JSON read with no schema).
- SP-13 (medium): a cache/persist inside a loop never unpersisted (low: a cached DataFrame used at most once afterwards).
- SP-14 (medium): RDD groupByKey.
- SP-15 (medium): a JDBC read with no partitionColumn/numPartitions/predicates.
- SP-16 (medium): a credential literal in a setting, a reader or writer option, a dict entry, a variable or a JDBC URL.
- SP-17 (low): orderBy/sort with no limit in the lineage of a write.
- SP-18 (low): show() or print(df.count()) outside loops.
- SP-19 (low): spark.executor.cores above 5, spark.executor.memory above 64 GB, spark.driver.maxResultSize 0.
- SP-20 (medium): mode("overwrite") with partitionBy and no dynamic partitionOverwriteMode.
- SP-21 (info): context - counts, the pyspark version, what is outside the check.

## Free extras

- Every read with its format, target, schema and JDBC partitioning; every write with its source, format, mode, partition columns and target; every action and whether it runs in a loop; every join with its key, type and whether it is broadcast; every Spark setting with its source.
- Put any audit finding (or all of them, or the fixed files) back into the paste and re-run the check; undo puts the original back.
- Exports: the check as Markdown, the flags as CSV, the edited or fixed job, the findings or changes as CSV, the whole result as JSON.

## Paid lanes (the `task` field)

- `audit`: `{task, verdict: blocked|needs_work|ready, headline, job, areas: [{area: driver|shuffle|compute|io, status: sound|fragile|broken, note}], findings: [{id, flags, file, line, severity, area, title, evidence, why, edits: [{before, after}]}], flag_answers: [{flag, status: confirmed|dismissed, reason}], plan, questions, summary}`. `before` is an exact slice of the file, applied in order.
- `fix`: `{task, verdict: fixed|partial, headline, files: [{path, content}], changes: [{file, function, flags, change, why}], verify, open: [{item, flags, reason}], questions, summary}`. `content` is the whole file; every function, read and write stays.
- Run input: `{task, code, facts, note?, audit_handoff?, retry_note?}` - all strings; `facts` is the page's analysis as a JSON string. The audit hands its findings to the fix lane with every edit applied to `code`.
- Model: gpt-terra, 10% publisher markup. A run needs a signed-in account; the reservation shown before a run is a hold, and only what the run uses is charged.

## Sources

- spark-optimization by Seth Hobson, wshobson/agents (github.com/wshobson/agents, plugins/data-engineering/skills/spark-optimization), MIT - https://skillsafe.ai/skill/@wshobson/spark-optimization
- The checks and the reconciliation are original to this app. Apache Spark and PySpark are trademarks of the Apache Software Foundation; this app is not affiliated with the ASF.