webpack issues

may need to run ‘sudo npm install’ first

install missing dependencies with `sudo npm install <DEP>`
“`
Module parse failed: /home/neil/Dropbox/w540-bak/bitbucket/sa-client-ui/src/app.js Unexpected token (199:40)
You may need an appropriate loader to handle this file type.
SyntaxError: Unexpected token (199:40)
“`

This seems to fix above: `sudo npm install –save-dev babel-loader` as mentioned in https://github.com/HenrikJoreteg/hjs-webpack

`Module not found: Error: Cannot resolve module ‘react’`

seems that packages have to be installed with `sudo nmp install …`

may need to run ‘sudo npm install’ first

Posted in Uncategorized | Leave a comment

DynamoDB: An AttributeValue may not contain an empty string (Exception)

Problem

com.amazonaws.AmazonServiceException: One or more parameter values were invalid: An AttributeValue may not contain an empty string (Service: AmazonDynamoDBv2;

Solution

This cryptic message basically says that dynamodb is not able to save fields with empty strings (or null) (a rather inconvenient and artificial limitation); but doesn’t say which ones …

This post suggest a related solution that works quite nicely.

implicit class RichJsObject(original: JsObject) {
  def omitEmpty: JsObject = original.value.foldLeft(original) { 
    case (obj, (key, JsString(st))) if st.isEmpty => obj - key
    case (obj, (key, JsArray(arr))) if arr.isEmpty => obj - key
    case (obj, (_, _)) => obj
  }
}

Then you can just apply it to your jsObj as: jsObj.omitEmpty

Often you may actually have a jsVallue; which could be converted to jsObj:

js.as[JsObject].omitEmpty.toString()

 

keywords: scala play json dynamodb empty null string insert

Posted in Uncategorized | Leave a comment

org.codehaus.mojo:nbm-maven-plugin: Module has friend dependency on but is not listed as a friend

Problem

[ERROR] Failed to execute goal org.codehaus.mojo:nbm-maven-plugin:3.14:manifest (default-manifest) on project move: Module has friend dependency on org.netbeans.bootstrap but is not listed as a friend. -> [Help 1]

Solution

Add scope: provided; e.g.:

<dependency>
<groupId>org.netbeans.modules</groupId>
<artifactId>org-netbeans-bootstrap</artifactId>
<version>RELEASE81</version>
<scope>provided</scope>
</dependency>

 

keywords: gephi netbeans maven 0.9

Posted in Uncategorized | Leave a comment

Upgrading Gephi 0.8 to 0.9

Gephi’s API had a rather significant API changes; here are some tips for the upgrades.

This nifty repo can get you started quickly: https://github.com/gephi/gephi-plugins

This repo provides good examples of API usage: https://github.com/gephi/gephi-toolkit-demos

API Changes

AttributeColumn subGraphFitnessCol = model.getNodeTable().getColumn(FLD_SUBGRAPH_FITNESS, AttributeType.DOUBLE);

->

Column subGraphCol = graphModel.getNodeTable().getColumn(FLD_GRAPH_FITNESS);

 

AttributeController ac = Lookup.getDefault().lookup(AttributeController.class);
AttributeModel model = ac.getModel();
AttributeColumn fitnessCol = model.getNodeTable().addColumn(“utility”, AttributeType.DOUBLE); // hack: add and then removed

->

graphModel.getNodeTable().

 

..

RankingController rankingController = Lookup.getDefault().lookup(RankingController.class);

// use sub/graph utility (depending on which graph is displayed)
Ranking ranking;
//AttributeModel attributeModel = Lookup.getDefault().lookup(AttributeController.class).getModel();
GraphModel attributeModel = graphModel;

if (jGraph.vertexSet().size() == this.superGraph.vertexSet().size()){
Column col = attributeModel.getNodeTable().getColumn(FLD_GRAPH_FITNESS);
ranking = rankingController.getModel().getRanking(Ranking.NODE_ELEMENT, col.getId() );
} else {
Column col = attributeModel.getNodeTable().getColumn(FLD_SUBGRAPH_FITNESS);
ranking = rankingController.getModel().getRanking(Ranking.NODE_ELEMENT, col.getId() );
}

AbstractSizeTransformer sizeTransformer = (AbstractSizeTransformer) rankingController.getModel().getTransformer(Ranking.NODE_ELEMENT, Transformer.RENDERABLE_SIZE);
sizeTransformer.setMinSize(10);
sizeTransformer.setMaxSize(40);
rankingController.transform(ranking, sizeTransformer);

=>

void size(boolean forceExecution) {
if (this.getIsSize() || forceExecution){
AppearanceController appearanceController = Lookup.getDefault().lookup(AppearanceController.class);
AppearanceModel appearanceModel = appearanceController.getModel();

// Get visible graph
GraphModel graphModel = Lookup.getDefault().lookup(GraphController.class).getGraphModel();
org.gephi.graph.api.Graph gGraph = graphModel.getGraphVisible();
// convert to jgrapht
Graph jGraph = JGraphTUtils.wrap(gGraph);
// update sizes

// use sub/graph utility (depending on which graph is displayed)
Column col;
//AttributeModel attributeModel = Lookup.getDefault().lookup(AttributeController.class).getModel();
GraphModel attributeModel = graphModel;

if (jGraph.vertexSet().size() == this.superGraph.vertexSet().size()){
col = attributeModel.getNodeTable().getColumn(FLD_GRAPH_FITNESS);
} else {
col = attributeModel.getNodeTable().getColumn(FLD_SUBGRAPH_FITNESS);
}

Function sizeRanking = appearanceModel.getNodeFunction(gGraph, col, RankingNodeSizeTransformer.class);
RankingNodeSizeTransformer sizeTransformer = (RankingNodeSizeTransformer) sizeRanking.getTransformer();
sizeTransformer.setMinSize(10);
sizeTransformer.setMaxSize(40);
appearanceController.transform(sizeRanking);

VizController.getInstance().getGraphIO().centerOnGraph();
}
}

 

Posted in Uncategorized | Leave a comment

Machine Learning Book Recommendations

I am often asked to recommend books on machine learning (ML).  Here is a WIP list of my recommendations: http://www.amazon.com/gp/registry/wishlist/1TLVK1WKJF47S/

 

Posted in Uncategorized | Leave a comment

Using AWS Lambda Alias with API Gateway

Using AWS lambda with specific alias through API Gateway; as described here with a command like:

aws lambda add-permission --function-name arn:aws:lambda:us-east-1:xxxxxx:function:getData --source-arn arn:aws:execute-api:us-east-1:xxxxxx:t463p6g84d/*/POST/data --principal apigateway.amazonaws.com --statement-id  --action lambda:InvokeFunction

Resulted in an error

aws: error: argument operation: Invalid choice, valid choices are:

The cause was the following could be due to the following:

The CLI command must be issued with credentials that have permission to call the “add-permission” action of the Lambda APIs.

However; I still couldn’t quite get it to work.  What did work; was adding `API Endpoint` from Lambda’s console; which assigns the necessarily permissions; seems that you can change endpoint later on and the permission will still work (I guess they are not attached to a specific endpoint).  Also don’t forget to re-deploy your API once you’ve made the changes.

 

 

keywords:

Add Permission to Lambda Function

You defined your Lambda function as a stage variable; you must manually give permissions to all the functions you will use. You can do this by running the below AWS CLI command for each function, replacing the stage variable in the function-name parameter with the necessary function name.

aws lambda add-permission –function-name arn:aws:lambda:us-east-1:1111111111:function:getSectorData:${stageVariables.lambdaAlias} –source-arn arn:aws:execute-api:us-east-1:1111111111:111/*/POST/data –principal apigateway.amazonaws.com –statement-id 111-4f1f-b5e5-111 –action lambda:InvokeFunction

 

 

Posted in Uncategorized | Leave a comment

AWS Lambda invoking from code

You can invoke a lambda from your code (details); you’d probably want to use the AWSLambdaAsyncClient.

CAUTION: you have to be very careful not to make re-cursive calls (unless that is your intention); otherwise you might get into an infinite loop.  Even worse if you are making several recursive calls you might exponentially increase the number of lambda instances that are running.

Here is a small snippet attempting to ensure that you are not making direct-recursive calls (note that you might still end up with indirectly recursive calls); use at your own risk; add the following to your handler function:

if (context.getFunctionName.equals(nextFunctionName)){
  throw new RuntimeException( context.getFunctionName + "is making a recursive lambda call")
}

 

 

 

keywords: call lambda from java scala code directly

 

 

 

Posted in Uncategorized | Leave a comment

DynamoDB: batch delete

Excerpt from my program; adapt as needed:

 

val client = new AmazonDynamoDBClient()
val dynamo = new DynamoDB(client)

val scanRequest = new ScanRequest()
  .withTableName(LocationRecord.TABLE_NAME)

val items = client.scan(scanRequest).getItems
items.grouped(25).foreach(group => {  // DynamoDB: Member must have length less than or equal to 25
  val delItems = new TableWriteItems(LocationRecord.TABLE_NAME)
  group.foreach(item => delItems.addPrimaryKeyToDelete(new PrimaryKey(LocationRecord.FLD_ID, item.get(LocationRecord.FLD_ID).getS)))
  dynamo.batchWriteItem(delItems)
})

 

see also: http://stackoverflow.com/questions/9154264/what-is-the-recomended-way-to-delete-a-large-number-of-items-from-dynamodb

keywords: aws dynamodb delete all records rows

Posted in Uncategorized | Leave a comment

does not map a @DynamoDBHashKey attribute; ensure a public, zero-parameter get method/field is annotated

Problem:

“`
Class does not map a @DynamoDBHashKey attribute; ensure a public, zero-parameter get method/field is annotated
com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMappingException: Class does not map a @DynamoDBHashKey attribute; ensure a public, zero-parameter get method/field is annotated
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMappingsRegistry$Mappings.getHashKey(DynamoDBMappingsRegistry.java:245)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper.needAutoGenerateAssignableKey(DynamoDBMapper.java:682)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper.save(DynamoDBMapper.java:708)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper.save(DynamoDBMapper.java:669)
“`

Solution:

Need to add `@beanGetter` to: `@(DynamoDBHashKey @beanGetter)`

e.g.: `@(DynamoDBHashKey @beanGetter)(attributeName=”ID”) @BeanProperty var ID: String = null`

Problem:

“`
The provided key element does not match the schema (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: )
com.amazonaws.AmazonServiceException: The provided key element does not match the schema (Service: AmazonDynamoDBv2; Status Code: 400; Error Code: ValidationException; Request ID: )
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1389)
at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:902)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:607)
at com.amazonaws.http.AmazonHttpClient.doExecute(AmazonHttpClient.java:376)
at com.amazonaws.http.AmazonHttpClient.executeWithTimer(AmazonHttpClient.java:338)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:287)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.doInvoke(AmazonDynamoDBClient.java:2000)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.invoke(AmazonDynamoDBClient.java:1970)
at com.amazonaws.services.dynamodbv2.AmazonDynamoDBClient.updateItem(AmazonDynamoDBClient.java:1798)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper$SaveObjectHandler.doUpdateItem(DynamoDBMapper.java:1095)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper$2.executeLowLevelRequest(DynamoDBMapper.java:795)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper$SaveObjectHandler.execute(DynamoDBMapper.java:974)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper.save(DynamoDBMapper.java:824)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper.save(DynamoDBMapper.java:669)
“`

Solution:

`attributeName` need to be provided for `DynamoDBHashKey` explicitly (even if it matches the field name).

e.g.: @(DynamoDBHashKey @beanGetter)(attributeName="ID") @BeanProperty var ID: String = null

 

 

com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMappingException: Class org.activeintel.al.crawler.LocationRecord does not map a @DynamoDBHashKey attribute; ensure a public, zero-parameter get method/field is annotated
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMappingsRegistry$Mappings.getHashKey(DynamoDBMappingsRegistry.java:245)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper.needAutoGenerateAssignableKey(DynamoDBMapper.java:682)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper.save(DynamoDBMapper.java:708)
at com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper.save(DynamoDBMapper.java:669)

Posted in Uncategorized | Leave a comment

AWS Lambda: extending execution time limit

Currently AWS Lambda limits execution time to 5 min.  However, in quite a few cases it is possible to overcome this limitation by simply re-invoking your function in an iterative manner.  Here are brief details.

You can keep a tab on remaining time with:   com.amazonaws.services.lambda.runtime.Context#getRemainingTimeInMillis

Once you are almost out of time you can simply re-invoke your own function directly from lambda (details); make sure to use the AWSLambdaAsyncClient and pass to your function whatever it needs to continue; e.g. resumptionPoint, intermediateResults etc.  And make sure you don’t get into an infinite loops (its going to be expensive).

 

 

 

keywords

Maximum execution duration per request 300 seconds

 

 

Posted in Uncategorized | 1 Comment