Skip to content

Commit d140f80

Browse files
committed
bloblib v0.0.3
0 parents  commit d140f80

40 files changed

+4855
-0
lines changed

.classpath

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<classpath>
3+
<classpathentry excluding="resources/" kind="src" output="target/classes" path="src/main/java">
4+
<attributes>
5+
<attribute name="optional" value="true"/>
6+
<attribute name="maven.pomderived" value="true"/>
7+
</attributes>
8+
</classpathentry>
9+
<classpathentry kind="src" output="target/test-classes" path="src/test/java">
10+
<attributes>
11+
<attribute name="optional" value="true"/>
12+
<attribute name="maven.pomderived" value="true"/>
13+
</attributes>
14+
</classpathentry>
15+
<classpathentry kind="con" path="org.eclipse.jdt.launching.JRE_CONTAINER/org.eclipse.jdt.internal.debug.ui.launcher.StandardVMType/JavaSE-1.7">
16+
<attributes>
17+
<attribute name="maven.pomderived" value="true"/>
18+
</attributes>
19+
</classpathentry>
20+
<classpathentry kind="con" path="org.eclipse.m2e.MAVEN2_CLASSPATH_CONTAINER">
21+
<attributes>
22+
<attribute name="maven.pomderived" value="true"/>
23+
</attributes>
24+
</classpathentry>
25+
<classpathentry kind="output" path="target/classes"/>
26+
</classpath>

.project

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<projectDescription>
3+
<name>bloblib</name>
4+
<comment></comment>
5+
<projects>
6+
</projects>
7+
<buildSpec>
8+
<buildCommand>
9+
<name>org.eclipse.jdt.core.javabuilder</name>
10+
<arguments>
11+
</arguments>
12+
</buildCommand>
13+
<buildCommand>
14+
<name>org.eclipse.m2e.core.maven2Builder</name>
15+
<arguments>
16+
</arguments>
17+
</buildCommand>
18+
</buildSpec>
19+
<natures>
20+
<nature>org.eclipse.jdt.core.javanature</nature>
21+
<nature>org.eclipse.m2e.core.maven2Nature</nature>
22+
</natures>
23+
</projectDescription>

README.md

Lines changed: 73 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
BlobLib
2+
=====
3+
BlobLib is the core libraries of the blobfs and blobfs-win, blobfs is a distributed [FUSE](http://fuse.sourceforge.net) based file system backed by [Microsoft azure blob storage service](https://azure.microsoft.com/en-us/services/storage/blobs/). It allows you to mount the containers/blobs in the storage account as a the local folder/driver. , no matter it is a Linux system or a Windows system. It support the cluster mode. you can mount the blob container (or part of it) across multiple linux and windows nodes. BlobLob can be used separately.
4+
5+
## Important Notes:
6+
* You can find the linux/Mac version of the [blobfs](https://github.com/wesley1975/blobfs).
7+
* You can find the windows version of the [blobfs-win](https://github.com/wesley1975/blobfs-win).
8+
* If you are interested in contributing, please contact me via jie1975.wu@gmail.com
9+
10+
## Project Goals
11+
Packaged the core operations of the azure blob storage as a separated library, this will make it easier for supporting different OS platform. This library can also be used alone with various projects.
12+
13+
14+
## Features:
15+
* Support man operations of the container,virtual directory and blob. such as create, delete, list, read, write etc.
16+
* New extension class of the InputStream and OutputStream. add a new cache layer and is optimized for the blob read and write.
17+
* Use blob leases as the distributed locking mechanism across multiple nodes. The blob will be locked exclusively when it is written.
18+
* The contents are pre-cached by chunks when there is read operation. This will eliminate the times of http request and increase the performance greatly.
19+
* Multi-part uploads are used for the write operation. Data is buffered firstly and then be uploaded if the buffer size exceed the threshold. This also can eliminate the times of http request and increase the performance greatly.
20+
* Append mode is supported, you can append the new line to the existing blob directly. this is more friendly for logging operation. And it can change the block blob to append blob automatically.
21+
* Use server-side copy for move, rename operations, more efficient for big files and folders.
22+
23+
## How To use
24+
### 1.include the bloblib in your project
25+
26+
### 2.read from the blob
27+
BlobReqParams insParams = new BlobReqParams();
28+
insParams.setContainer("orders");
29+
insParams.setBfsBlobType(BfsBlobType.BLOCKBLOB);
30+
insParams.setBlob("neworders.log");
31+
BlobBufferedIns bbIns = new BlobBufferedIns(insParams);
32+
String line;
33+
while ((line = bbIns.readLine()) != null)
34+
{
35+
System.out.println((line));
36+
Thread.sleep(100);
37+
}
38+
### 3.Write to the blob
39+
BlobReqParams ousParams = new BlobReqParams();
40+
ousParams.setContainer("orders");
41+
ousParams.setBfsBlobType(BfsBlobType.BLOCKBLOB); // use BfsBlobType.APPENDBLOB to create a append blob
42+
ousParams.setBlob("neworders.log");
43+
BlobBufferedOus bbOus = new BlobBufferedOus(ousParams);
44+
String line = "insert new line here";
45+
bbOus.writeLine(line);
46+
bbOus.close();
47+
48+
### 4.please find other functions from the source code.
49+
50+
51+
It is highly recommended that you should config it as a windows services.
52+
53+
## Performance Test
54+
* The performance depends on the machine and the network.
55+
56+
## Dependency
57+
* [azure-storage](https://github.com/Azure/azure-storage-java): Microsoft Azure Storage Library for Java .
58+
59+
## Limitation and known issues:
60+
* For the file copy, the blobfs will use read out - then write in to new blob mode. this will spent more time for large files/folders.
61+
* For the page blob, currently, should be, but it is not well tested yet. it may casue file interruption.
62+
63+
## Supported platforms
64+
-Linux
65+
-MacOS
66+
-windows
67+
68+
## License
69+
Copyright (C) 2017 Wesley Wu jie1975.wu@gmail.com
70+
This code is licensed under The General Public License version 3
71+
72+
## FeedBack
73+
Your feedbacks are highly appreciated! :)

bin/blobfs.conf

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# ================================================================================
2+
# ++++++++++++++++++ blobfs configurations +++++++++++++++++++++++++++++++++++++++
3+
# ================================================================================
4+
# Azure storage account connection string
5+
Storage_Connection_String = DefaultEndpointsProtocol=https;AccountName=dev2107;AccountKey=wJNRBizfsyQ2JUBBBJjBFAsxjBmvVpkK+61ZqKV/n3HaI/jSEdS3oGoyBGdmvBhVVeUy5S2hlGSAuipeyXXSww==;EndpointSuffix=core.chinacloudapi.cn
6+
7+
# the name of the queue, which will be used for synchronizing the file cache across the nodes.
8+
queue_name = blobfs
9+
10+
# the prefix of the blobs that will be used as the mounted blobfs root,
11+
# e.g., /container1/blob1/; defaults to /
12+
blob_prefix = /
13+
14+
# Desired local mount point for BlobFs, linux and osx
15+
mount_point = /mnt/blobfs
16+
17+
# Desired local mount point for BlobFs, windows
18+
win_mount_point = Y:\
19+
20+
# the user id for the BlogFs, default value is the caller user id
21+
# default value is :-1.
22+
uid = -1
23+
24+
# the group id for the BlogFs, default value is the group id of the caller
25+
# default value is :-1.
26+
gid = -1
27+
28+
29+
# In the Debug Mode, the debug messages will be displayed in the console, currently this option is ignored.
30+
debug_enabled = true
31+
32+
# supports five logging levels: TRACE < DEBUG < INFO < WARNING < ERROR.
33+
log_level = ERROR
34+
35+
# cache
36+
cache_enabled = true
37+
38+
# in seconds
39+
cache_TTL = 180
40+
41+
# if one source of the blobs will be mounted by more than one host, you should enable the cluster mode
42+
# in the cluster mode, blobfs will manage the cache distributedly
43+
cluster_enabled = true
44+
45+
# change the block blob to append blob automatically,
46+
# this will be triggered when you open a read only file with the append mode.
47+
# caution for large blob, this will consume more time
48+
# e.g., echo "new line" >> readonlyfile
49+
auto_change_block_blob_to_append_blob = true

bin/bloblib-0.0.3.jar

1.62 MB
Binary file not shown.

blobfs.conf

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# ================================================================================
2+
# ++++++++++++++++++ blobfs configurations +++++++++++++++++++++++++++++++++++++++
3+
# ================================================================================
4+
# Azure storage account connection string
5+
Storage_Connection_String = DefaultEndpointsProtocol=https;AccountName=dev2107;AccountKey=wJNRBizfsyQ2JUBBBJjBFAsxjBmvVpkK+61ZqKV/n3HaI/jSEdS3oGoyBGdmvBhVVeUy5S2hlGSAuipeyXXSww==;EndpointSuffix=core.chinacloudapi.cn
6+
7+
# the name of the queue, which will be used for synchronizing the file cache across the nodes.
8+
queue_name = blobfs
9+
10+
# the prefix of the blobs that will be used as the mounted blobfs root,
11+
# e.g., /container1/blob1/; defaults to /
12+
blob_prefix = /music2017/
13+
14+
# Desired local mount point for BlobFs, linux and osx
15+
mount_point = /mnt/blobfs
16+
17+
# Desired local mount point for BlobFs, windows
18+
win_mount_point = Y:\\
19+
20+
# the user id for the BlogFs, default value is the caller user id
21+
# default value is :-1.
22+
uid = -1
23+
24+
# the group id for the BlogFs, default value is the group id of the caller
25+
# default value is :-1.
26+
gid = -1
27+
28+
29+
# In the Debug Mode, the debug messages will be displayed in the console
30+
debug_enabled = true
31+
32+
# supports five logging levels: TRACE < DEBUG < INFO < WARNING < ERROR.
33+
log_level = INFO
34+
35+
# cache
36+
cache_enabled = true
37+
38+
# in seconds
39+
cache_TTL = 180
40+
41+
# if one source of the blobs will be mounted by more than one host, you should enable the cluster mode
42+
# in the cluster mode, blobfs will manage the cache distributedly
43+
cluster_enabled = true
44+
45+
# change the block blob to append blob automatically,
46+
# this will be triggered when you open a read only file with the append mode.
47+
# caution for large blob, this will consume more time
48+
# e.g., echo "new line" >> readonlyfile
49+
auto_change_block_blob_to_append_blob = true

dependency-reduced-pom.xml

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
3+
<modelVersion>4.0.0</modelVersion>
4+
<groupId>com.wesley</groupId>
5+
<artifactId>bloblib</artifactId>
6+
<name>A blob operation library built on top of azure blob service</name>
7+
<version>0.0.3</version>
8+
<url>http://maven.apache.org</url>
9+
<build>
10+
<sourceDirectory>src</sourceDirectory>
11+
<resources>
12+
<resource>
13+
<directory>src/main/resources</directory>
14+
</resource>
15+
</resources>
16+
<plugins>
17+
<plugin>
18+
<artifactId>maven-compiler-plugin</artifactId>
19+
<version>3.6.1</version>
20+
<configuration>
21+
<source>1.7</source>
22+
<target>1.7</target>
23+
</configuration>
24+
</plugin>
25+
<plugin>
26+
<artifactId>maven-jar-plugin</artifactId>
27+
<version>3.0.2</version>
28+
<configuration>
29+
<archive>
30+
<manifest>
31+
<mainClass>com.wesley.bloblib.BlobLib</mainClass>
32+
</manifest>
33+
</archive>
34+
</configuration>
35+
</plugin>
36+
<plugin>
37+
<artifactId>maven-shade-plugin</artifactId>
38+
<version>3.0.0</version>
39+
<executions>
40+
<execution>
41+
<phase>package</phase>
42+
<goals>
43+
<goal>shade</goal>
44+
</goals>
45+
<configuration>
46+
<transformers>
47+
<transformer>
48+
<mainClass>com.wesley.bloblib.BlobLib</mainClass>
49+
</transformer>
50+
<transformer />
51+
</transformers>
52+
<filters>
53+
<filter>
54+
<artifact>*:*</artifact>
55+
<excludes>
56+
<exclude>META-INF/*.SF</exclude>
57+
<exclude>META-INF/*.DSA</exclude>
58+
<exclude>META-INF/*.RSA</exclude>
59+
</excludes>
60+
</filter>
61+
</filters>
62+
</configuration>
63+
</execution>
64+
</executions>
65+
<configuration>
66+
<finalName>${project.artifactId}-${project.version}</finalName>
67+
</configuration>
68+
</plugin>
69+
</plugins>
70+
</build>
71+
<properties>
72+
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
73+
<java.version>1.7</java.version>
74+
<maven.compiler.source>1.7</maven.compiler.source>
75+
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
76+
<maven.compiler.target>1.7</maven.compiler.target>
77+
</properties>
78+
</project>
79+

0 commit comments

Comments
 (0)