Spark repartition

repartitionByRange

---

repartitionByRange(numPartitions, *cols) method of pyspark.sql.dataframe.DataFrame instance
    Returns a new :class:`DataFrame` partitioned by the given partitioning expressions. The
    resulting DataFrame is range partitioned.
    
    :param numPartitions:
        can be an int to specify the target number of partitions or a Column.
        If it is a Column, it will be used as the first partitioning column. If not specified,
        the default number of partitions is used.
    
    At least one partition-by expression must be specified.
    When no explicit sort order is specified, "ascending nulls first" is assumed.

begin = time.time()
df = merge_data
df.repartitionByRange(10,"probeset_id").write.format("delta").mode("append").save(f)
print(time.time()-begin)