Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
pyspark.sql.utils.AnalysisException: Found duplicate column(s) when inserting into ...
As the error message says, there are duplicated columns in your Spark SQL code.
Fix …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
py4j.protocol.Py4JJavaError: An error occurred while calling o156.getQuotaUsage.
org.apache.hadoop.ipc.RemoteException(java.io.IOException): The quota system is disabled in Router.
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
Caused by: java.lang.RuntimeException: Could not find any configured addresses for URI hdfs://clustername-router/
This is due to missing clustername-router settings in the property dfs.nameservices in …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: hdfs::/cluster-name/user/dclong/feature_example/features/train/2022-03-11
As the error message points out, there's a syntax …
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
LightGBM support pandas columns of category type.
As a matter of fact,
this is the suggested way of handling categorical columns in LightGBM.
data[feature] = pd.Series(data[feature], dtype="category")
A LightGBM model (which is a Booster object) records categories of each categorical feature. This information is used to set categories of each categorical feature during prediction, which ensures that a LightGBM model can always handle categorical features correctly.
Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!
A deque is implemented via the circular queue data structure and it has O(1) time complexity appending from both ends.
Unlike list and tuple collections, a deque CANNOT be sliced!