Commits

Breadcrumbs

History for

onmaster

User selector

All users

All time

Commit History

Commits on May 14, 2025

• [SPARK-52126][SS][PYTHON] Revert rename for TWS utility classes for forward compatibility in Spark Connect

  Show description for 0922233
  HeartSaVioR

  committed
  0922233

  Copy full SHA for 0922233

• [SPARK-52114][PYTHON][TESTS] Test UDF evaluation type under useArrow with session-level Arrow config

  Show description for 7b22a9b
  

  xinrong-meng

  authored and

  zhengruifeng

  committed
  7b22a9b

  Copy full SHA for 7b22a9b

• [SPARK-52112][PYTHON][TESTS] Refactor UDF profiling tests with shared assertion utility

  Show description for 2f7cd6e
  

  xinrong-meng

  authored and

  zhengruifeng

  committed
  2f7cd6e

  Copy full SHA for 2f7cd6e

Commits on May 13, 2025

• [SPARK-52092][PYTHON][TESTS] Add UDF tests for empty DataFrames

  Show description for 14dd441
  xinrong-meng

  committed
  14dd441

  Copy full SHA for 14dd441

• [SPARK-52022][PYTHON][FOLLOW-UP] Explicitly check thrown exceptions

  Show description for 5f5d818
  HyukjinKwon

  committed
  5f5d818

  Copy full SHA for 5f5d818

• [SPARK-50585][PYTHON][DOCS] Render interactive PySpark plot examples

  Show description for a57b8d7
  xinrong-meng

  committed
  a57b8d7

  Copy full SHA for a57b8d7

• [SPARK-46685][DOCS] Refine PySpark profiler user guide for clarity

  Show description for d67208a
  

  xinrong-meng

  authored and

  HyukjinKwon

  committed
  d67208a

  Copy full SHA for d67208a

Commits on May 12, 2025

• [SPARK-52077][PYTHON][CONNECT][TEST] Skip ArrowUDTFParityTests in Spark Connect compatibility test

  Show description for dfc8175
  HyukjinKwon

  committed
  dfc8175

  Copy full SHA for dfc8175

• [SPARK-52076][CONNECT] Explicitly closes ExecutePlanResponseReattachableIterator after usage

  Show description for dcac3f5
  HyukjinKwon

  committed
  dcac3f5

  Copy full SHA for dcac3f5

• [SPARK-52057][ML][CONNECT] Collect Tree size limit warning messages to client

  Show description for b50690a
  WeichenXu123

  committed
  b50690a

  Copy full SHA for b50690a

• [MINOR][PYSPARK][SQL] fixed a typo in python/pyspark/sql/column.py

  Show description for 4788bae
  

  francesco-camaione

  authored and

  HyukjinKwon

  committed
  4788bae

  Copy full SHA for 4788bae

Commits on May 9, 2025

• [SPARK-52055][PYTHON][CONNECT] Suppress the exception in ExecutePlanResponseReattachableIterator.__del__

  Show description for 10c7614
  

  HyukjinKwon

  authored and

  dongjoon-hyun

  committed
  10c7614

  Copy full SHA for 10c7614

• [SPARK-52051][ML][CONNECT] Enable model summary when memory control is enabled

  Show description for 568f920
  WeichenXu123

  committed
  568f920

  Copy full SHA for 568f920

• [SPARK-51789][CORE][FOLLOW-UP] Set the initial Spark Connect mode properly

  Show description for 207e296
  

  HyukjinKwon

  authored and

  cloud-fan

  committed
  207e296

  Copy full SHA for 207e296

• [SPARK-52053][PYTHON][TESTS] Enable test_job_cancellation and its parity test

  Show description for da2b449
  zhengruifeng

  committed
  da2b449

  Copy full SHA for da2b449

• [SPARK-52047][PYTHON] Raise PySparkValueError for unsupported plot kinds

  Show description for 9831552
  

  xinrong-meng

  authored and

  HyukjinKwon

  committed
  9831552

  Copy full SHA for 9831552

• [SPARK-51983][PS][TESTS][FOLLOW-UP] Replace `assertEquals` with `assertEqual` for python 3.12+

  Show description for 3bfe059
  zhengruifeng

  committed
  3bfe059

  Copy full SHA for 3bfe059

• [SPARK-52013][CONNECT][ML] Remove `SparkConnectClient.ml_caches`

  Show description for edc6312
  WeichenXu123

  committed
  edc6312

  Copy full SHA for edc6312

• [SPARK-52040][PYTHON][SQL][CONNECT] ResolveLateralColumnAliasReference should retain the plan id

  Show description for 688281a
  

  zhengruifeng

  authored and

  cloud-fan

  committed
  688281a

  Copy full SHA for 688281a

Commits on May 8, 2025

• [SPARK-51981][SS] Add JobTags to queryStartedEvent

  Show description for 271cd0d
  

  Lingkai Kong

  authored and

  HyukjinKwon

  committed
  271cd0d

  Copy full SHA for 271cd0d

• [SPARK-52026][PS] Block pandas API on Spark on ANSI mode by default

  Show description for 584f468
  

  ueshin

  authored and

  cloud-fan

  committed
  584f468

  Copy full SHA for 584f468

Commits on May 7, 2025

• [SPARK-51974][CONNECT][ML] Limit model size and per-session model cache size

  Show description for ca77925
  WeichenXu123

  committed
  ca77925

  Copy full SHA for ca77925

• [SPARK-52022][PYTHON] Add additional check in default method for SparkThrowable.getQueryContext

  Show description for 39f6d25
  HyukjinKwon

  committed
  39f6d25

  Copy full SHA for 39f6d25

Commits on May 6, 2025

• [MINOR][TESTS] Reduce the number of iterations in tuning tests

  Show description for 9e7bfc5
  

  zhengruifeng

  authored and

  HyukjinKwon

  committed
  9e7bfc5

  Copy full SHA for 9e7bfc5

• [SPARK-51983][PS] Prepare the test environment for pandas API on Spark with ANSI mode enabled

  Show description for 56be8b0
  

  ueshin

  authored and

  HyukjinKwon

  committed
  56be8b0

  Copy full SHA for 56be8b0

• [SPARK-44856][PYTHON] Improve Python UDTF arrow serializer performance

  Show description for b957d7f
  HyukjinKwon

  committed
  b957d7f

  Copy full SHA for b957d7f

• [SPARK-51989][PYTHON] Add missing Filter subclasses to __all__ list in datasource

  Show description for 633419a
  allisonwang-db

  committed
  633419a

  Copy full SHA for 633419a

• [SPARK-51182][SQL] DataFrameWriter should throw dataPathNotSpecifiedError when path is not specified

  Show description for b02afd8
  

  vrozov

  authored and

  cloud-fan

  committed
  b02afd8

  Copy full SHA for b02afd8

Commits on May 5, 2025

• [SPARK-50238][PYTHON][FOLLOW-UP] Fix Arrow UDF bug where Struct with metadata field was failing on the is_variant path

  Show description for 4d4fdc0
  

  harshmotw-db

  authored and

  ueshin

  committed
  4d4fdc0

  Copy full SHA for 4d4fdc0

• [SPARK-51947] Spark connect model cache offloading

  Show description for 894d828
  WeichenXu123

  committed
  894d828

  Copy full SHA for 894d828

Commits on May 4, 2025

• [MINOR][PS][DOC] Update pandas API on Spark option doc

  Show description for e857f43
  

  ueshin

  authored and

  HyukjinKwon

  committed
  e857f43

  Copy full SHA for e857f43

Commits on Apr 30, 2025

• [SPARK-51959][PYTHON] Make functions not import ParentDataFrame

  Show description for 13902f5
  

  zhengruifeng

  authored and

  HyukjinKwon

  committed
  13902f5

  Copy full SHA for 13902f5

Commits on Apr 29, 2025

• [MINOR][PYTHON][TESTS] Clean up a group of connect tests

  Show description for 54eb1a2
  zhengruifeng

  committed
  54eb1a2

  Copy full SHA for 54eb1a2

• [SPARK-51931][SQL] Add maxBytesPerOutputBatch to limit the number of bytes of Arrow output batch

  Show description for 86bf4c8
  viirya

  committed
  86bf4c8

  Copy full SHA for 86bf4c8

• [SPARK-51802][PYTHON][DOCS] OSS PySpark User Guide Docs

  Show description for 02faae5
  

  asl3

  authored and

  HyukjinKwon

  committed
  02faae5

  Copy full SHA for 02faae5