Category Archives: Programming

Splunk HTTP Event Collector Python 3 Example

With Splunk’s latest release of version 6.3 a new feature called HTTP Event Collector has been added. It allows for sending JSON formatted data to Splunk via an HTTP call. I won’t go into all the details of this feature in this post, but for the curious more information can be found here.

This feature is great for anyone who wants to easily get data into Splunk using their own scripts. With this being a new feature there is not yet many examples of how to use this on the scripting side. In this post I want to provide an example in Python that others can use to build upon in their own code.

Below is a short and documented example using the urllib library to craft an HTTP request that Splunk’s HTTP Event Collector will accept.

import urllib.request
import json

def send_event(splunk_host, auth_token, log_data):
   """Sends an event to the HTTP Event collector of a Splunk Instance"""
   
   try:
      # Integer value representing epoch time format
      event_time = 0
      
      # String representing the host name or IP
      host_id = "localhost"
      
      # String representing the Splunk sourcetype, see:
      # docs.splunk.com/Documentation/Splunk/6.3.2/Data/Listofpretrainedsourcetypes
      source_type = "access_combined"
      
      # Create request URL
      request_url = "http://%s:8088/services/collector" % splunk_host
      
      post_data = {
         "time": event_time, 
         "host": host_id,
         "sourcetype": source_type,
         "event": log_data
      }
      
      # Encode data in JSON utf-8 format
      data = json.dumps(post_data).encode('utf8')
      
      # Create auth header
      auth_header = "Splunk %s" % auth_token
      headers = {'Authorization' : auth_header}
      
      # Create request
      req = urllib.request.Request(request_url, data, headers)
      response = urllib.request.urlopen(req)
      
      # read response, should be in JSON format
      read_response = response.read()
      
      try:
         response_json = json.loads(str(read_response)[2:-1])
         
         if "text" in response_json:
            if response_json["text"] == "Success":
               post_success = True
            else:
               post_success = False
      except:
         post_success = False
      
      if post_success == True:
         # Event was recieved successfully
         print ("Event was recieved successfully")
      else:
         # Event returned an error
         print ("Error sending request.")
      
   except Exception as err:
      # Network or connection error
      post_success = False
      print ("Error sending request")
      print (str(err))

   return post_success

def main():
   splunk_auth_token = "00000000-0000-0000-0000-000000000000"
   splunk_host = "10.11.12.13"
   
   log_data = {
      "data_point_1": 50,
      "data_point_2": 20,
   }
   
   result = send_event(splunk_host, splunk_auth_token, log_data)
   print (result)

main()

A few things to note: this example is not using SSL, so the Enable SSL check box in the HTTP Event Collector global settings must be unchecked. Also Splunk is picky about the top level JSON keys, only a few specific keys can be used. Those keys are: time, host, source, sourcetype, index and event. All custom data should be under the event key. Finally this code should work in all versions of Python after 3.0.

Photo Upload and Preview with Paper.js

With the rise of social media and collaborative sites, users are sharing more of their own content than ever. While users upload many different types of content to these sites, for this post I am focusing on photos.

Chances are if you code for the web you have or will have to write code to allow users to upload their photos. In the old days you could get away with a simple file selection and upload. However now days’ users expect to preview their images and to be able to use a few simple editing features on the photo before uploading it. In this post I write about photo rotation and cropping, but depending on the context they may expect more. The more might include tools like photo filters or user tagging.

Before starting this project, I Googled a bit to see what was already out there. While I found quite a few examples and libraries to show an image preview, none of those allowed any editing of the photo before upload. I feel that giving the user some basic editing ability is important, and what’s more I think users expect to have that ability on modern websites. So with that idea in mind I began creating a photo upload and preview widget with these basic tools.

I didn’t want to rewrite a graphics library as plenty of those already exist, but I would have to choose one for this project. I’ve used Paper.js for other projects in the past and my familiarity with it is the sole reason I chose it for this project. I’m sure any other similar library would work just as well, especially since this project does not call for any complex graphical work. That being said, Paper.js also has other raster features that would make implementing future improvements like photo filters an easy task.

For this project I started by building the HTML and CSS to structure the layout. The photo tool icons are SVG images that can be styled by CSS. For photo section I used a standard file input. The change event on this input tigers a custom function that adds the image to the Paper.js canvas and activates the edit controls. For the photo itself I didn’t want to affect its resolution by resizing the image, so when the image is added I use the view zoom to fit the photo in the viewport. This preserves the original photo size and resolution.

Rotation was easy as there was built in functions to do this. The hard part was dealing with cropping the image after rotation. Paper.js does not return image dimensions correctly when that image is rotated. The position and dimensions are those of the non-rotated image. To compensate I had to write code to rotate the given position and dimensions.

After the rotation issue solved, I then had to extract the image as a dataURL and craft an AJAX request. That finished up the client side part of the code. As a proof of concept I created a quick php receiving file. It takes the given dataURL and creates an image file on the server. It’s just a skeleton but could be easily adapted to most projects.

I’ve created a repository on GitHub for this project for interested readers. Also check out the client side code on this CodePen:

See the Pen Upload Image and Preview by Alec Dhuse (@alecdhuse) on CodePen.

Basic SVG+CSS Downloading Animation

Since finding out about SVG graphics years ago, I have been a fan. I love how they look on high resolution screens compared to raster graphics. Until now, I have not had the chance to try out using them for animations. One of my projects required an animation to show an item that is in the process of being downloaded. Rather then using an animated GIF, I opted to go with an animated SVG.

I wanted something that would look familiar to users as a loading icon. So I started with a basic completing circle animation. I also wanted to add a bit more context to the animation so I added a downward arrow and a horizontal line. My hope is that users will connect this symbol with the action of downloading something. I have shown the icon to a few people they had no trouble understanding that it indicated a downloading item. A wider audience may disagree, but feel free to judge for yourself.

Now on to the technical aspect. SVG animations require that the SVG XML be embedded in the document not added as a linked file from an image tag. For a simple image, like mine, this is easy to do without half the document being image code. For a more complex drawing this won’t be the case and I won’t get into that here.

I created the SVG elements in Inkscape and coped the raw image instruction for the elements out of the file using a text editor. I then placed the SVG instructions within a G tag to group the separate objects and make it possible to easily transform my new icon.

The next piece is the CSS. I started by creating three selectors that allow me set three states of my icon: not downloaded, downloading and downloaded. They each set the icon’s color and play state.

#download {
   stroke:#000;
   fill: #000;
}

#downloaded {
   stroke:#00bbe0;
   fill: #00bbe0;
   -webkit-animation-play-state: paused;
   animation-play-state: paused;
}

#downloading {
   stroke:#00bbe0;
   fill: #00bbe0;
}

Next we create a selector for circle outline that gives instructions for hiding part of the the completed circle. To create the completing circle animation we will use a transparent dash stroke and change it’s offset to show a varied completeness of the circle.

The stroke-dasharray value has to be longer than the circumference of the outer circle. I chose a value of 500 so if the whole animation was made a bit larger, the animation would still work as designed. The animation value is how long the transition between empty to full will take to complete. I chose two seconds as it seems to be a comfortable speed.

Since this is an animation a keyframe selector must be specified. In this example the keyframe selector has two sub-selectors: to and from. These specify the starting and ending states of the animation. Combined with the stroke-dashoffset declaration we specify the starting position of our transparent dash. For this I am transitioning from a larger value to a smaller one, to make the circle fill clockwise. Swapping the values makes the animation go in reverse. For the current size of the drawing a difference of about 100 is required.

Below you can see the final product

See the Pen Basic SVG+CSS Downloading Animation by Alec Dhuse (@alecdhuse) on CodePen.


A Few More Notes:

Compatibility of the CSS declarations for animations is mixed, which leads to many duplicate declarations in order to make animations cross browser compatible.

Second, is that if the CSS specifies specific animation declarations and the general animation declaration, the more specific must be placed after the general. Like this:

...
 animation: show 2s;
 animation-iteration-count: infinite;
...

Folding Map goes open source!

Screenshot of the Folding Map application.

Screenshot of the Folding Map application.

Folding Map is a project that I have been working on for several years.  I’ve used it to create maps of villages in Cameroon, trail maps in the Pacific NorthWest and most recently for climbing maps on my new site TopoHawk.

It’s come a long ways since its humble beginnings and now it’s time to share my work with the world.  While it doesn’t have the features of QGIS it works well to create simple maps, visualizations or to export for use on the web.

The source code is available from GitHub so check it out.  Issue tracking will also be handled there so feel free to request features or report problems or contribute to the project.

Java GeoJSON Reader

I have recently started working on a Java based GeoJSON reader.  It has the ability to read plain JSON, but this version creates Java objects for the various GeoJSON objects defined in the GeoJSON specification: coordinate, point, line, polygon, etc.  This makes importing and using GeoJSON in a Java program much easier.

I’ve just committed an initial working version and I’m planning to update it in the near future.  I would like to eventually add an easy way to build the GeoJSON objects and output them to a file as well.

Check out the project on GitHub.