Ben Chuanlong Du's Blog

It is never too late to learn.

Use LightGBM With Spark

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

https://github.com/Azure/mmlspark/blob/master/docs/lightgbm.md

MMLSpark seems to be the best option to use train models using LightGBM on a Spark cluster. Note that MMLSpark requires …

The Python Quit Unexpectedly Error

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

If you are using Homebrew, you can reinstall a corrupted Python environment like this:

brew uninstall --ignore-dependencies --force python python@2 unset PYTHONPATH brew install python python@2 I had another …

Gradient Clipping in PyTorch

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

optimizer.zero_grad()        
output = model(data)
loss = F.nll_loss(output, target)
loss.backward()
torch.nn.utils.clip_grad_norm_(model.parameters(), args.clip)
optimizer.step()
  1. Use torch.nn.utils.clips_grad_norm_ (which is in-place) instead …

Tensor Transformations in TorchVision

Comments

  1. Transformations in torchvision.transforms work on images, tensors (representing images) and possibly on numpy arrays (representing images). However, a transformation (e.g., ToTensor) might work differently on different input types. So you'd be clear about what exactly a transformation function does. A good practice is to always convert your non-tensor input data to tensors using the transformation ToTensor

Cluster Management Made Easy with the Python Package Fabric

Things on this page are fragmentary and immature notes/thoughts of the author. Please read with your own judgement!

Ansible is a better alternative to Fabric. It is suggested that you use Ansible instead.

  1. Docstring will be displayed when you type the command fab -l.

  2. Invoke is for local use …