Download files from a URL – Python script

Downloading files from a web server over the internet is a trivial task which is performed in our everyday life. Isnt it?

For example when accessing a web site, your browser downloads some files to your computer. Commonly, it could obviously be .html files and then it might include .css .js files as well. You dont notice it as it is handled and managed by the browser itself. However there can be instances that you need to access a specific URL and download some files. No doubt it can be done manually. What if there are hundreds of files to be downloaded. In this article it is discussed how to get it done with minimal effort using a Python script.

Disclaimer – Employing a script to access/ download content from a web server can cause a higher traffic towards the web server. Therefore do not use this knowledge to access any resource in a web server without getting prior permissions. Use this script at your own risk. The author bears no responsibility.

For the purpose we are going to use the library urllib which is an inbuilt library in Python. .urlopen and .read methods can be used to access the content of the resource and then .write method can be used to save the content to the local machine. Check how the code is organized.

Python Version 2.7.15

import urllib
url = 'http://url-for-resources'

webf = urllib.urlopen(url)
txt = webf.read()
f = open('local-machine-folder-path', 'w+')
f.write(txt)
f.close()

It is pretty straight forward. Then the next objective is to automate the process. Using a loop is the easiest. One issue to overcome though. How to get the file names. If the file names follow a regular sequence for example data_01.json, data_02.json, data_03.json etc, it is possible to generate the sequence. Otherwise file names should be inserted to the code. A solution could be to extract the file names to a file and then read the file and get filenames. 

Here is the Python script for the above approach.


import urllib

f = open("local-file-with-file-names", "r")
file_list = f.readlines()


for x in file_list:

	filename = x.strip()
	url = 'http://url-for-resources/'+ filename

	webf = urllib.urlopen(url)
	txt = webf.read()
	f = open('local-file-path-to-save/'+filename, 'w+')
	f.write(txt)
	f.close()
	
	print('Saving file ' + filename)

One line contains one file name in the file which is used to include files to be downloaded.

Advertisements

Constitutionality of the NCM – Democracy As Seen By A Dummy

Trias Politica or Separation of power is a common concept among the democratic hemisphere. The basic concept holds the governance of the state. Being a Democratic Republic state, the governing structure of Sri Lanka comprises of three branches. namely the Executive, the Legislature and the Judiciary. These are three independent institutions which are governed by the supreme law of the land which is currently the 1978 Constitution 19th Amendment. How these three entities should adhere to the governance of the state is clearly mentioned in the constitution. In basic definition:

The Executive which is lead by the Executive President who is the head of the state and of the government, is responsible for implementing and administering the laws passed by the legislature.

The Legislature which is the Parliament, is responsible for enacting the laws of the state.

The Judiciary which is the Supreme Court, is responsible for interpreting the Constitution and the laws enacted by the legislature.

As these three institutions are independent and their duties are mutually exclusive, the Constitution has provided required regulatory powers to safeguard the continuity of the system, to be specific the balance of the democracy. Due to these boundaries no institution subdues another and ultimately ensures these entities are following the Constitution. This protective measures prevents the tyranny among the institutions which is an inherent weakness of democracy as the classical Greek philosopher Plato discussed.

One of such key important limitations imposed by the Constitution is the power of interpreting the Constitution. In fact the legislature has no right to interpret the Constitution even though the legislature is the body which enacts it. This is clearly mentioned in the Article 125, Section 1 of the Constitution. Any matter which arises the constitutionality should be brought before the Supreme Court.

125. (1) The Supreme Court shall have sole and exclusive jurisdiction to hear and determine any question relating to the interpretation of the Constitution and accordingly, whenever any such question arises in the course of any proceedings in any other court or tribunal or other institution empowered by law to administer justice or to exercise judicial or quasi-judicial functions, such question shall forthwith be referred to the Supreme Court for determination. The Supreme Court may direct that further proceedings be stayed pending the determination of such question.

Therefore at this point, it is clear that any assertions of breaching the Constitution should be adjudicated by the Supreme Court, neither the Executive nor the Legislature is privileged to hold a proclamation.

Below is the proposal for the No Confidence Motion (NCM) submitted to the parliament yesterday (2018 November 14) by the JVP[1].

යෝජනාව

මෙම ගරු සභාව,

  1. අතිගරු ජනාධිපතිවරයා විසින් ප්‍රකාශයට පත්කල 2018 ඔක්තෝබර් 26 වන දින හා අංක 2094/43, 2094/43 A සහ 2094/44 දරණ අති විශේෂ ගැසට් නිවේදන සහ එහි සදහන් නියෝගයන් සහ පත්කිරීම් ආණ්ඩුක්‍රම ව්‍යවස්ථාවට පටහැනි වන බවත්, නීතිය ඉදිරියේ ශුන්‍ය සහ බල රහිත බවත්, නීතිය ඉදිරියේ කිසිදු බලයක් හෝ වලංගු භාවයක් නොමැති බවත්,
  2. ඉහත සදහන් හේතු නිසා 2018 ඔක්තෝබර් 26 වන දා හෝ ඉන් පසුව හෝ පත් කල අග්‍රාමාත්‍ය මහින්ද රාජපක්ෂ මහතා හා කැබිනට් මණ්ඩලය හා එලෙස පත් කරන ලද්දාවූ ආණ්ඩුව කෙරෙහි කිසිදු විශ්වාසයක් නොමැති බවත්,

යෝජනා කර සිටී.

In the above proposal, it lacks the adjudication of the Supreme Court but supported by an assertion of the proposers. According to the Article 125 section 1 of the Constitution, nobody interprets the Constitution but the Supreme Court. In such grounds the proposal is merely based on personal interests of some MPs, not to mention the majority of 122 MPs. Therefore the proposal itself has no legal standing form the inception and cannot be voted for or against. The proposal is null and void. And specially the MPs were violating the Article 125 section 1 of the Constitution as they were testifying their own assertions and including it in a proposal to the house. In the contatary the speaker has accepted an incomplete document which lacks constitutional adherence. Both are serious misconducts and setting adverse examples.

The proper procedure was to seek the Supreme Court opinion over the incident and present the NCM which upholds the democracy as laid by the supreme law of the state.

References

[1] http://www.dailymirror.lk/article/JVP-hands-over-no-faith-motion-against-PM-Rajapaksa-158322.html

Attribution – The featured image used in this article is attributed to Vikalpa | Groundviews | Maatram | CPA from Sri Lanka, courtesy of https://commons.wikimedia.org. This file is licensed under the Creative Commons Attribution 2.0 Generic license.

Disclaimer – The writer is neither an expert on Constitutional law nor a counsellor at law. The content is based on the observations of the states political status quo and the personal opinion of the writer.

Copyright Waiver No Copyright, This article was released in Public Domain except the featuring image in this article. This work has been identified as being free of known restrictions under copyright law, including all related and neighboring rights. You can copy, modify, distribute and perform the work, even for commercial purposes, all without asking permission.

Inoculation the Balance of the Democracy

Countries in which Presidential system is in place, enjoy the separation of powers. Democratic Socialist Republic of Sri Lanka is one of such states. Basically the branches of separation are the Executive, the Legislature and the Judiciary. The executive president enjoys number of privileges as the head of the state and the head of the government. One of them is the immunity over the jurisdictions [Article 35 of the Constitution]. It is vested by the constitution itself. At the same time, the constitution proclaims that nobody is above the law [Article 12 Section 1 in the Constitution].

If the facts hold as above, then how can the Supreme Court which is the Judicial branch of the separation of power can go against the executive presidency.

In order to understand the dynamics, the Article 35 of the Constitution should be closely examined.  The Executive president cannot be indicted for civil or criminal proceedings as long as he/ she holds the office. Even though it grants a supremacy over the presidency, the constitution itself places few boundaries. According to the Article 126 Section 1 of the Constitution, the Supreme Court possesses the jurisdiction against the matters which infringe the fundamental rights established by the supreme law of the state which is the Constitution [Chapter III and IV of the Constitution]. In fact this clause resides since the inception of the 1978 Constitution. The root of the infringement could be an executive or administrative action. According to the Article 126 Section 5 of the Constitution the hearing should be concluded within 2 months.

This is how the executive presidency has been protected, as a result the president is not capable of ruling against the Constitution.

President’s orders are often termed as Executive Orders, well famoused in the United States of America. The Supreme Court had issued restraining orders for Trump’s Executive Order on travel policies at the beginning of his term is one of the recent examples. Recently CNN reports that they are planning to sue the President for banning Jim Acosta[1]. 

In Sri Lanka, the extraordinary gazette no 2096/70, issued for dissolving the Parliament is being contested before the Supreme Court[2]. In such a background, the writer finds the statements of the judicial supremacy/ immunity of the executive presidency is fatuous. The Supreme Court has the absolute powers to make sure the executive is within the boundaries of the supreme law of the state. Challenging the order issued by the president before the Supreme Court is the righteous move.

References

[1] https://edition.cnn.com/2018/11/13/media/cnn-sues-trump/index.html
[2] https://www.aljazeera.com/news/2018/11/sri-lanka-parties-petition-court-parliament-dissolution-181112072846303.html
[3] http://www.dailymirror.lk/article/Court-has-no-jurisdiction-to-hear-FR-petitions-against-dissolution-AG-158282.html

Disclaimer : The sole objective of this article is discussing the balance of the democracy hence the purpose is purely educational. This neither a discussion of an ongoing proceeding nor an opinion on it.

Who is to be accepted as the premier?

Recent political developments in Democratic Socialist Republic of Sri Lanka is perplexed. Many misleading ideas are being planted by the conventional and social media. One of the biggest illusions are the status quo of the premiership.

The power of governing the state has been vested on three pillars, the executive, the legislature and the jurisdiction. These three independent organizations are governed [1] by the prime law of the state which is the 1978 constitution which has been amended recently in 2015 (the 19th amendment).

According to the current constitution, the legislature which is the Parliament or the executive which is the president reserves the right of appointing or removing the prime minister given that the laid conditions are met in the constitution. The jurisdiction has no power over this nevertheless the supreme court is able to adjudicate on any matter related to the constitution.

On such grounds, on 26th October 2018, the executive has issued two extraordinary gazettes.

Gazette No 2094/43 – Removal of the prime minister – Ranil Wickramasinghe [2]

Gazette No 2094/44 – Appointment of the prime minister – Mahinda Rajapaksa [3]

Neither the jurisdiction has adjudicated the appointment nor the parliament has ruled otherwise yet. Therefore contemporary prime minister in Democratic Socialist Republic of Sri Lanka is Mahinda Rajapaksa. No country has the right to reject or approve that decision as it directly affects the sovereignty of the state.

[1] – Article 4 (a) (b) (c) The Constitution – 19th Amendment of the 1978 constitution
[2] – http://www.documents.gov.lk/files/egz/2018/10/2094-43_E.pdf
[3] – http://www.documents.gov.lk/files/egz/2018/10/2094-44_E.pdf

Node-to-Node Secure Data Transmission Protocol for Low-power IoT Devices

This paper has been presented in 2018 International Conference on Advances in ICT for Emerging Regions (ICTer) held in Colombo on 2018 September 28th.

Node-to-Node Secure Data Transmission Protocol for Low-power IoT Devices

Authors : Chandradeepa Kurera (University of Moratuwa, Katubedda, Sri Lanka.), Dasun Navoda

Session Chair : Prof. Christian Rohner

Abstract— Through the internet and local networks, IoT devices exchange data. Most of the IoT devices are low-power devices, meaning that they are designed to use less electric power. To secure data transmission, it is required to encrypt the messages. Encryption and decryption of messages are computationally expensive activities, thus require considerable amount of processing and memory power which is not affordable to low-power IoT devices. Therefore, not all secure transmission protocols are low-power IoT devices friendly. This study proposes a secure data transmission protocol for low-power IoT devices. The design inherits some features in Kerberos and one-time password concepts. The protocol is designed for devices which are connected to each other, as in a fully connected network topology. The protocol uses symmetric key cryptography under the assumption of that the device specific keys are never being transmitted over the network. It resists DoS, message replay and Man-of-the-middle attacks while facilitating the key security concepts such as Authenticity, Confidentiality and Integrity. The designed protocol uses less number of encryption/ decryption cycles and maintain session based strong authentication to facilitate secure data transmission among nodes.

Keywords— IOT, Secure Data Transmission, Node to node data transmission, Transmission Protocol, Low-power IoT devices

Corresponding Author – bckurera AT gmail DOT com

Conference Web Site – http://www.icter.org/

Published in IEEE Xplorehttps://doi.org/10.1109/ICTER.2018.8615511 

Copy of the Paper can be found at – https://www.researchgate.net/publication/330474676_Node-to-Node_Secure_Data_Transmission_Protocol_for_Low-power_IoT_Devices

Ballerinalang – Union Type

Ballerinalang is a strongly typed language, meaning that type of a variable is very important. As a result, if a variable is defined as an int type variable, it cannot hold any other value which does not belong to the said data type, such as a float type value.  Then in functions in Ballerinalang, it is possible to return multiple values which belongs to different types, discussed earlier in Ballerinalang – Functions post. When a function can return values belong to multiple types, there should be a way of assigning those different values with different types. Should not it?

The concept is as simple as the name, it explains itself. In a union type, the variable can take any value which belong to it’s component types. Ok, lets say a union type variable is defined using types int, float and string. Then that variable can hold an int type value (integer) or a float type value (decimal figure) or a string (some characters). But at a given time it can take one form, value which belongs to a type, only one type.

Following is the syntax:

type1 | type2….. | type n <variable name>;

or

type1 | type2….. | type n <variable name> = <initialization value> ;

Define Union Type variable

Union type variable can have many types. The implicit default value is ( ). Check the following code:

import ballerina/io;

function main(string... param){
    int|float|string one_var;
    float|boolean|() two_var = 3.14;

    io:println("Value of one_var = ", one_var);
    io:println("Value of two_var = ", two_var);

    one_var = "Assigning string value";
    two_var = false;

    io:println("Value of one_var = ", one_var);
    io:println("Value of two_var = ", two_var);
}



Output
Value of one_var =
Value of two_var = 3.14
Value of one_var = Assigning string value
Value of two_var = false

Observe the following scenario which captures a value passed by a function which returns multiple types.

import ballerina/io;
import ballerina/math;

function main(string... param){
    float|string converted_speed;

    converted_speed = convert(45.5);
    io:println("Converted Speed is - ", converted_speed);

    converted_speed = convert(100);
    io:println("Converted Speed is - ", converted_speed);
}

function convert(float speed) returns (float|string){
    if(speed < 0) || (speed>60) {
        return "Not defined !";
    }
    return speed * 0.45;
}



Output
Converted Speed is - 20.475
Converted Speed is - Not defined !

The converted_speed variable can take two type of values, float and string. Therefore based on the type the function returns, the variable hold the value.

More…

Actually, the variable can take many forms but it is not purely belonging to one type. For an example, at any time, it is wrong to identify one_var as a string type variable, because it holds a string value. It is type is compuse always. All time, the type of variable one_var is int|float|string

In order to fully understand the logic, observe the following example:

import ballerina/io;
import ballerina/math;

function main(string... param){
    float|string speed = "NA";

    if (speed=="NA"){
        speed = math:random() * 60.0;
    }

    io:println("Current speed = ",speed);

    speed = speed * 0.5;
    io:println("Half of Current speed = ",speed);
}

Output
operator '*' not defined for 'float|string' and 'float'
compilation contains errors

If following line is excluded, the program works fine. Because speed is not considered as a float type value though it holds a float type value. It is always float|string type variable.

speed = speed * 0.5;

In case it is required to carry out the above calculation, then it is required to define another float type variable, get the calculation done and assign the value to the speed variable. That should be the correct way of doing it.

import ballerina/io;
import ballerina/math;

function main(string... param){
    float|string speed = "NA";

    if (speed=="NA"){
        speed = math:random() * 60.0;
    }

    io:println("Current speed = ",speed);

    float half_speed = speed;
    half_speed = half_speed * 0.5;
    speed = half_speed;

    io:println("Half of Current speed = ",speed);
}


Output
incompatible types: expected 'float', found 'float|string'
compilation contains errors

Then another error, it is obvious right. Because since the newly defined variable is float type, then it is not possible to assign float|string type value. Then what is the way around here?

This is why match construct and match expression come handy. Will discuss them in the next post.

Ballerinalang – Maps

In case it is required to maintain an array with defined indexes, Maps is the right choice in Ballerina. Not like arrays Maps are mutable, meaning that once the variable has been defined it is not possible to modify it. In maps indexes are termed as fields.

Defining a Map

When defining a map variable, the type of the map can be added. However it is not mandatory. The default type, if no type is mentioned, taken as any. The value of fields of the map, should belong to string type.


import ballerina/io;

function main(string... param){
    map person_info = {"name": "John", "age": 25, "isMarried": true};
    io:println(person_info);

    map speed = {"pointA": 12.75, "pointB": 15.0, "pointC": 5.658};
    io:println(speed);
}

Output
{"name":"John", "age":25, "isMarried":true}
{"pointA":12.75, "pointB":15.0, "pointC":5.658}

Note that when a map is defined with no type, any value can be assigned to it’s filed, whereas when a map is defined with a type, it is not possible to assign value with other types.

To access values in a map,the dot (.) operator or field name can be used.


import ballerina/io;

function main(string... param){
    map person_info = {"name": "John", "age": 25, "isMarried": true};
    io:println(person_info.name , " is ", person_info.age, " years old.");

    map speed = {"pointA": 12.75, "pointB": 15.0, "pointC": 5.658};
    io:println(speed["pointC"]);
}

Output
John is 25 years old
5.658

Observe how map is used in the following example, hope it helps to understand the usability of maps.


import ballerina/io;

function main(string... param){
    map vehicle = {"carType":"A", "mass": 1500, "speed": 60, "isSpeedMetric": false, "unit":0};
    
    io:println("Before Conversion = ",vehicle);

    vehicle.unit = "km/h";       

    if(! check vehicle.isSpeedMetric){
       vehicle.speed = convertToMetric(vehicle); 
       vehicle.isSpeedMetric = true;
    }    
    io:println("After Conversion = ",vehicle);
    io:println("Speed in Metric Units = ",vehicle.speed," ",vehicle.unit);
}

function convertToMetric(map vehicle_info) returns float{
    if(check vehicle_info.isSpeedMetric){
        return check vehicle_info.speed;
    }
    var speed = check vehicle_info.speed;
    return speed*1.60934;
}

Output
Before Conversion = {"carType":"A", "mass":1500, "speed":60, "isSpeedMetric":false, "unit":0}
After Conversion = {"carType":"A", "mass":1500, "speed":96.5604, "isSpeedMetric":true, "unit":"km/h"}
Speed in Metric Units = 96.5604 km/h

Type conversion and type checks need to be performed as the map is not defined with a type.