Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Total size of serialized results is bigger than spark.driver.maxResultSize
Eliminate unnecessary broadcast or collect.
If one of the tables for joining contains too large number of partitions …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
GRANT
priv_type [, priv_type ] ...
ON database_table_or_view_name
TO principal_specification [, principal_specification] ...
[WITH GRANT OPTION];
Examples:
GRANT SELECT ON table1 TO USER user1;
GRANT SELECT ON DATABASE db1 TO USER user1 …Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Excessive logging is better than no logging! This is generally true in distributed big data applications.
Use loguru if it is available.
If you have to use the logging module,
be …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
The Docker image dclong/jupyterhub-toree has Spark and Apache Toree installed and configured. Since Spark is already installed in it, you don't need to download and install Spark by yourself. By …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
The split-by-leaf mode (grow_policy="lossguide") is not supported in distributed training,
which makes XGBoost4J on Spark much slower than LightGBM on Spark.
https://towardsdatascience.com/build-xgboost-lightgbm-models-on-large-datasets-what-are-the-possible-solutions-bf882da2c27d
https://xgboost …