無為閣

為無為,事無事

Serialized Python Function Object

想法

利用marshal把python bytecode轉成string, 就可以pickled.

把Function轉成Pickled string

1
2
3
4
5
6
7
8
import pickle
import marshal
maintask_bytecode = marshal.dumps(maintask_callback.func_code)
output = {
  'maintask_bytecode': maintask_bytecode,
  'maintask_defaults': maintask_defaults,
  'maintask_args': maintask_args}
print pickle.dumps(output)

把Pickled string轉成function

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
import sys
import types
import marshal
import pickle
## \brief reconstruct function from bytecode
#  \param func_bytecode str marshaled bytecode of func_code
#  \param func_default dict func_defaults
#  \return Function
def reconstruct_function(func_bytecode, func_defaults):
    func_code = marshal.loads(func_bytecode)
    return types.FunctionType(func_code, globals(), func_code.co_name,
                             func_defaults)
string = sys.stdin.read()
received = pickle.loads(string)
callback = reconstruct_function(received['maintask_bytecode'],
      received['maintask_defaults'])
print callback(maintask_args)

Fix Chromium Can Not Play Mp4

I found my chromium in Ubuntu 11.10 can not play mp4 video. after I googling, the solution is to install chromium-codecs-ffmpeg-extra to solve these problems.

$ apt-get install chromium-codecs-ffmpeg-extra

How to Create a Custom Launchpad Bugs Report

I already use lp-cli to generate reports about one month and think this may benefit someone has the same requirmentation. so here you are.

This document will show

  • howto collected bugs data with custom search conditions
  • combine collected bugs data and report template

comments are welcome.

Metaclass in Python (1) - Singleton

Singleton 這東西大家想必不陌生,在Python裡實做的方式滿多, 這邊用這個當例子介紹 Metaclass 可以做什麼。

概念上很簡當, 讓Class的建構子不能產生Instance, 然後再提供一個 Classmethod 能夠取得Instance(而且只能有一個), 在Python你可以用id來檢查兩個物件 是不是相同的。

原則上Singleton我只會用在不用花腦袋就會覺得這個class建出來的instance這個程式執行 時只能有一個。

Singleton 的源碼

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
class Singleton(type):
    def __init__(cls,name,bases,dic):
        super(Singleton,cls).__init__(name,bases,dic)
        cls.instance=None

    def __call__(cls, *args, **kwargs):
      # 這裡不raise Exception, 是因為doctest比較好寫
            print "please use get_instance function to get the instance"
            # 你也可讓cls()直接傳回instance, 讓class user不用在意他用的class
      # 是不是Singleton, 他只要注意class的主功能即可
            # return cls.get_instance(*args, *kw)

    def get_instance(cls,*args,**kw):
        if cls.instance is None:
            cls.instance=super(Singleton,cls).__call__(*args,**kw)
        return cls.instance

用法

把你想變成Singleton class 的 metaclass 設成 Singleton 就可以了, 後悔的話,把那一行註解起來,這個class就不是Singleton。

要注意的地方是這class的Singleton特性是可以被繼承的,但這也是為什麼我喜歡用 這種方式的原因,另一個好處是Class的功能跟Design Pattern的耦合度會比較低。 缺點就是會遇到metaclass衝突的狀況,但也不是不能解決。

1
2
class _db(object):
  __metaclass__ = Singleton

完整的源碼加測試

(metaclass_Singleton.py) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
#!/usr/bin/env python
# vim: tabstop=4 expandtab shiftwidth=4 softtabstop=4
# -*- encoding=utf8 -*-
#
# Author 2012 Hsin-Yi Chen
class Singleton(type):
    def __init__(cls,name,bases,dic):
        super(Singleton,cls).__init__(name,bases,dic)
        cls.instance=None

    def __call__(cls, *args, **kwargs):
            print "please use get_instance function to get the instance"

    def get_instance(cls,*args,**kw):
        if cls.instance is None:
            cls.instance=super(Singleton,cls).__call__(*args,**kw)
        return cls.instance

class _db(object):
    """
    >>> obj=_db()
    please use get_instance function to get the instance
    >>> obj is None
    True
    >>> obj1 = _db.get_instance()
    connecting to 0
    >>> obj2 = _db.get_instance()
    >>> id(obj1) == id(obj2)
    True
    """
    __metaclass__ = Singleton
    session_max = 0

    def __init__(self):
        print 'connecting to {0}'.format(self.session_max)

class MySQL(_db):
    """
    >>> obj1 = MySQL.get_instance()
    connecting to 1000
    >>> obj2 = MySQL.get_instance()
    >>> id(obj1) == id(obj2)
    True
    """
    session_max = 1000

import doctest
doctest.testmod()

Will You Attend PyCon TW 2012?

從兩三年前開始,就不斷有人在喊”那會有Python Conference 嗎”?

尤其是去年有Ruby Con TW 2011, PHP Conf TW 2011

終於,前陣子PyCon TW 2012 官方網站公佈啦! 活動預計在2012 年 6月9日(六)及10日(日) 假中研院人文館舉辦, 目前正在統計可能參與人數, 請幫忙填問卷

  • 有興趣提供講題的人,請到這裡
  • 有興趣發表論文的人,請到這裡
  • 有興趣提供贊助的公司,請到這裡

用Python的人應該是不少才對, 大家不要在潛水了!

How-To: Change string.template’s Delimiter in Runtime

這篇文章提到Template因為用到了Metaclass, 因此改Class或是Instance的delimiter是沒作用的, 因為Template的pattern atrtibute 在 class 建構時, 就已經決定好, 於是你只能建立一個class 來繼承他, 把Template.delimiter Overwrite掉, 這樣 Template().delimter 才是你要的, 可我實在懶得為了改delimiter就去寫一個自定義class, 原文的方式我也覺得有點麻煩, 因為要組出完整的pattern

基本上這很好解, 就runtime時產生一個自定義class就好了 :p

1
2
3
4
5
6
7
8
9
10
   import string
  def create_tpl(content, **tpl_config):
          if tpl_config:
                  tplcls = type('CustomTemplate', (string.Template,), tpl_config)
          else:
                  tplcls = string.Template
          return tplcls(content)

  tpl = create_tpl('#aa', delimiter='#')
  print tpl.substitute(aa=1)

How-to: 輸出已被redirect 到時, 想要把字串印到Terminal

最近在寫lp-cli時, 因為stdout已經用來傳遞pickle data到下一個command, 例子如下

# 找出ossug-hychen' 正在處理幾個bug
$ lp-searchbugs people ossug-hychen --status 'In Progress' | lp-print count

所以如果我想要print 一些資訊到terminal 其實是辦不的, 昨日在H4時與Thinker討論時, 他提出了一個辦法

如果要在輸出已被redirect 到時, 想要把字串印到Terminal,可以把訊息寫到 /dev/tty (等於目前的 control terminal)

1
2
3
4
5
6
7
        # Use TTY for print information or debugging message
        # Use STDOUT for printing MissionMessage
        self.tty_fd = os.open('/dev/tty', os.O_WRONLY)
          self.bk_stdout = os.dup(sys.stdout.fileno())
        os.close(sys.stdout.fileno())
        os.dup(self.tty_fd)
        self.bk_stdout = os.fdopen(self.bk_stdout, 'w')

但是這方法仍有缺點:

  1. stdout 沒辦法被rederiect,
  2. 不能用cron, 因為cron沒有tty

Meeting Room Name

台北辦公室擴建完畢後, 內部在詢問會議室的命名意見,命名規則很簡單。要有台灣特色且容易翻譯。 於是想當然爾眾人的提案不外乎高山、景點、小吃之類的。

因會議室剛好有午間,有同事提議用台灣五院來命名,而且這樣以後辦公室會有下列有趣的對話….

同事A: 等下在哪開會

同事B: 立法院

當然這個提案,不是很被多數人認可,因為英文名字有點太長…

而我自己的提案是台灣高山湖, 這些湖的共同點就是交通不便,需要親力親為上山後方能領略美景。 至於象徵意含嘛… 我沒想這麼多。

五個會議室名稱分別如下

  1. 嘉明湖 - 海拔3310/11公尺, 素有天使的眼淚之稱 七彩湖 - 海拔2980公尺

  2. 松蘿湖 - 海拔1300公尺, 素有「十七歲之湖」及「夢幻之湖」之稱

  3. 七彩湖 - 海拔2980公尺

  4. 白石池 - 海拔2770公尺

  5. 翠池 - 海拔3520公尺 , 台灣最高的高山湖泊

註:我還有另一個提案是用台灣36秘境中的5個來命名, 不過目前比較偏好用高山湖泊命名

Lp-cli - Just Another Command Line Tool of Launchpad

The web interface of Launchpad is awesome, but it does not provide a function that you can save your search result, or even create a custom bug display list, I believe managers would like this function, especially they want to track each team member status.

I am just a engineer but also have almost the same requirement, here are 3 questions I need to answer in the end of weekend.

  • Which bugs I worked this week?
  • Which bugs I need to work next week?
  • Which bugs are waiting me to fix?

It is better to have a way to always show bugs I am interesting for different purpose, so I start to write a script by using python-launchpadlib.

by the way, another needs is to find bugs and modify attribute(s) of each bug.

python-launchpad

Launchpad has a very friendly python library, I will not talk this too much, if you want to know the details of it, please check these websites.

lp-cli

lp-cli include several tools which is very small and has only one trivial purpose, that is users can use Unix pipeline to combine small tools to archive a complex task.

All work of process data of Launchpad can be simplified as the following

  1. get data from Launchpad
  2. get sub set of data by some conditions
  3. do something on each entry one or multiple times
  4. do something on the final result such as save as a file or print to console

Use Case 1

Create a text file contains bugs I worked is modified this week.

1
2
3
4
$ lp-searchbugs people ossug-hychen  | \
  lp-view modified_thisweek | \
  lp-print buglist |  \
  lp-save bugs_modified_thisweek.txt
  • lp-searchbugs people ossug-hychen – find related bugtask of a user that login name is ossug-hychen
  • lp-view modified_thisweek – get bugs worked by ossug-hychen before and modified this week
  • lp-print buglist – then print the result in buglist style
  • lp-save bugs_worked_this_week.txt – then save the result as a file

the view modified_thisweek means the bug worked by anyone since 7 days before as defined here

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
---
data_getters:
...
        modified_thisweek:
                status: [Fix Committed,
                         Fix Released,
                         In Progress,
                         Incomplete (with response),
                         Incomplete (without response),
                         Triaged,
                         Invalid,
                         Won't Fix,
                         Confirmed]
                modified_since: $date_before_now
...

so if you want to know which bugs is really worked by yourself, we need another tool to limit the scope of result which is not done yet.

Use Case 2

reassign bugs I fixed in project ABC to another people, some time is QA

1
2
$ lp-searchbugs project ABC --status 'Fix Committed' --assignee ossug-hychen  | \
  lp-edit --assignee QA
  • first command is to get bugs fixed by me in project ABC
  • second command is to reassign each bugs to a QA guy

Available Commands

  • lp-searchbugs - type:Fetcher, generate a task of searching bugs in launchpad
  • lp-view - type:View, add extra args to received task (like lp-searchbugs)
  • lp-print - type:Task Action, execute received task and print the result
  • lp-save - type:Action, save STDIN to a file
  • lp-mybugs - shortcut of lp-searchbugs people $your-lp-login-id

because accessing Launchpad takes a lot of time, so the real data fetching work is only executed in Task Action command in my design.

[update]

Tim Chen told me a cool idea I never think about that you can save STDOUT of lp-searchbugs , lp-mybugs or lp-view to a text file

1
$ lp-mybugs | lp-view plan_nextweek > bugs_I_need_to_work_next_week.view

next time when you want to get the result, just type

1
$ lp-print buglist <  bugs_I_need_to_work_next_week.view

this behavior is similar what lp-view does, so now you can create any view you want!

Installation

1
2
3
$ add-apt-repository ppa:ossug-hychen/ppa
$ apt-get update
$ apt-get install lp-cli

Thanks

The idea of lp-cli is inspired by my friend Thinker Li’s image downloading tool

Decorator of Display Deprecated Warring in Python

用途: 用來警告 function 或 class method 已過時,如果有指定取代的function的話,在runtime時改用取代的function

1
2
3
4
5
6
7
8
9
Python 2.7.2+ (default, Oct  4 2011, 20:03:08)
[GCC 4.6.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> from vsgui.api import input_text
>>> input_text
<function ask_text at 0x8e0009c>
>>> input_text('input it')
vsgui/api.py:45: DeprecationWarning: Call to deprecated function input_text.; use ask_text instead
  def input_text(*args, **kwargs):

用法一 - 只宣告function已被depreacted, 不指定取代的function

1
2
3
4
5
6
@depreacted() # <- 這一段表示執行deprecated() 取得wrap function
def old1():   # 當old1()被呼叫時,實際上是先呼叫wrap(old1)以取得old1 function, 再執行old()
  print 1

print 'function is '+old1
old1()

執行結果:

1
2
3
4
function is <function old1 at 0xb7317f44>
recipe-deprecated-warring.py:45: DeprecationWarning: Call to deprecated function old1.
  def old1():
called old1

因為有使用functools.wraps,所以雖然old1 這個變數實際上已繫結在wrap function, 但print時還是會顯示為old1 function

用法二: 除了警告外,還會在runtime時改執行取代depreacted function的新functoin

1
2
3
4
5
@depreacted(new1) # <- 這一段表示執行deprecated(new1) 取得wrap
def old2():       # 當old1()被呼叫時,實際上是呼叫wrap(old1) 取得new_func 再執行 new_func
  print 1   # 而new_func會再執行new1,並把*args, **kwargs全數pass給new1

print 'function is '+old2

執行結果

1
2
3
4
<function new1 at 0xb7292dbc>
source/downloads/code/recipe-deprecated-warring.py:49: DeprecationWarning: Call to deprecated function old2.; use new1 instead
  def old2():
called new1

注意: 目前還不能對class method 指定 replacement method

源始碼

重用了Python Wiki - Smart deprecation warnings , Active Code Stack - deprecated 的部份源碼

(recipe-deprecated-warring.py) download
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
# took codes from the following:
#
#- [Python Wiki - Smart deprecation warnings ](http://wiki.python.org/moin/PythonDecoratorLibrary#Smart_deprecation_warnings_.28with_valid_filenames.2C_line_numbers.2C_etc..29)
# - [Active Code Stack - deprecated ](http://code.activestate.com/recipes/391367-deprecated/)

import os
import warnings
import functools

# enable to show warring
warnings.simplefilter('default')

def deprecated(replacement=None):
    """This is a decorator which can be used to mark functions
    as deprecated. It will result in a warning being emitted
    when the function is used.

    ref:
        - recipe-391367-1 on active stack code
        - recipe-577819-1 on active stack code

    @replacement function replacement function
    """
    def wrapper(old_func):
        wrapped_func = replacement and replacement or old_func
        @functools.wraps(wrapped_func)
        def new_func(*args, **kwargs):
            msg = "Call to deprecated function %(funcname)s." % {
                    'funcname': old_func.__name__}
            if replacement:
                msg += "; use {} instead".format(replacement.__name__)
            warnings.warn_explicit(msg,
                category=DeprecationWarning,
                filename=old_func.func_code.co_filename,
                lineno=old_func.func_code.co_firstlineno + 1
            )
            return wrapped_func(*args, **kwargs)
        return new_func
    return wrapper

def new1():
    print 'called new1'

@deprecated()
def old1():
    print 'called old1'

@deprecated(new1)
def old2():
    print 'called old1'

if __name__ == '__main__':
    print old1
    old1()
    print old2
    old2()

運作原理

  • 第一個decorator用來產生第二個decorator wrapper
  • 第二個decorator wrapper 用來產生新的function new_func
  • 執行新的function new_func 會先顯示warrning, 最後再把參數傳給要wrapped_func 最後再傳回wrapped_func執行結果
  • 在runtime 時,因為是閉包的關係, 要執行的 wrapped_func 根據decorator wrapper的replacement參數來決定是用來代換的function, 還是原本的function.

下面是抽象化後的源碼

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
@deprecated()
def _sum(args):
  pass

def deprecated(replacement=None): # 第一個 decorator
  ....
  def real_decorator(original_func): # 第二個 decorator,
                     # 套用在original function 上, 也就是_sum
      ....
      def func_with_warn(*args, **kwargs): # 產生好新function
          ....
          # 顯示warring字串
          ....
          return original_func(*args, **kwargs) # 回傳_sum的執行解果
      return func_with_warn
  return real_decorator

參考資料