LocalStack の S3 を使う場合の注意点¶

概要¶

LocalStack は AWS サービスをローカルに構築して、本物の代わりに使えるようにするテスト環境です。

使ってみる¶

この docker-compose.yml で起動してみます。

docker-compose run main

エンドポイントを LocalStack に向けることでアクセスします。

aws --endpoint=http://localstack:4566 s3 ls

S3 エンドポイントについて¶

S3 のエンドポイントにはバーチャルホスト形式(virtual hosted-style)とパス形式(path-style)が存在します。パス形式は廃止の方向のようですが、スケジュールについては未定です(2021/05現在)。

形式	例
バーチャルホスト形式	`https://(BucketName).s3.(Region).amazonaws.com`
パス形式	`https://s3.(Region).amazonaws.com`

CLI や SDK で S3 バケットにアクセスした場合、デフォルトではバーチャルホスト形式にしたエンドポイントへ要求します。LocalStack では http://(BucketName).localhost.localstack.cloud:4566/(ObjectKey) という形になります。

対策 1 : パス形式を使用する。¶

https://github.com/localstack/localstack/issues/3943

@whummer Thank you for your advice.

I tried using endpoint http://s3.localhost.localstack.cloud:4566, but the bucket is not created. It's fine with use_path_style_endpoint, but I prefer not to use it if possible because of deprecation.

How to reproduce¶

Create docker-compose.yml.

version: '3'
services:
  localstack:
    image: localstack/localstack:0.12.10
    environment:
      - DEFAULT_REGION=ap-northeast-1
      - SERVICES=s3
      - DEBUG=1
    hostname: localstack
    networks:
      default:
        aliases:
          - s3.localhost.localstack.cloud
          - test.s3.localhost.localstack.cloud
          - test.test.s3.localhost.localstack.cloud
  main:
    image: php:7.4.18-cli
    links:
      - localstack:localstack

Run containers.

docker-compose run main bash

CreateBucket (success, but bucket not created.)

$s3 = new Aws\S3\S3Client([
    'version' => '2006-03-01',
    'region' => 'ap-northeast-1',
    'endpoint' => 'http://s3.localhost.localstack.cloud:4566',
]);
$s3->createBucket([
    'Bucket' => 'test.test',
]);

CreateBucket (successfully created.)

$s3 = new Aws\S3\S3Client([
    'version' => '2006-03-01',
    'region' => 'ap-northeast-1',
    'endpoint' => 'http://s3.localhost.localstack.cloud:4566',
    'use_path_style_endpoint' => true,
]);
$s3->createBucket([
    'Bucket' => 'test.test',
]);

PutObject failure¶

After I created test.test bucket with use_path_style_endpoint, I tried upload object to endpoint http://s3.localhost.localstack.cloud:4566, but 500 error occured.

$s3 = new Aws\S3\S3Client([
    'version' => '2006-03-01',
    'region' => 'ap-northeast-1',
    'endpoint' => 'http://s3.localhost.localstack.cloud:4566',
]);
$s3->putObject([
    'Bucket' => 'test.test',
    'Key' => 'test.txt,
    'SourceFile' => 'test.txt',
]);

Stack trace is:

2021-05-02 23:06:52,870:API: 127.0.0.1 - - [02/May/2021 23:06:52] "PUT /test.txt HTTP/1.1" 500 -
2021-05-02 23:06:52,894:API: Error on request:
Traceback (most recent call last):
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/werkzeug/serving.py", line 323, in run_wsgi
    execute(self.server.app)
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/werkzeug/serving.py", line 312, in execute
    application_iter = app(environ, start_response)
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/moto/server.py", line 177, in __call__
    return backend_app(environ, start_response)
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/flask/app.py", line 2464, in __call__
    return self.wsgi_app(environ, start_response)
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/flask/app.py", line 2450, in wsgi_app
    response = self.handle_exception(e)
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/flask/app.py", line 1867, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/flask_cors/extension.py", line 165, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/moto/core/utils.py", line 156, in __call__
    result = self.callback(request, request.url, {})
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/moto/s3/responses.py", line 256, in ambiguous_response
    return self.bucket_response(request, full_url, headers)
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/moto/s3/responses.py", line 265, in bucket_response
    response = self._bucket_response(request, full_url, headers)
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/moto/s3/responses.py", line 311, in _bucket_response
    return self._bucket_response_put(
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/moto/s3/responses.py", line 773, in _bucket_response_put
    if self._create_bucket_configuration_is_empty(body):
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/moto/s3/responses.py", line 645, in _create_bucket_configuration_is_empty
    create_bucket_configuration = xmltodict.parse(body)[
  File "/opt/code/localstack/.venv/lib/python3.8/site-packages/xmltodict.py", line 327, in parse
    parser.Parse(xml_input, True)
xml.parsers.expat.ExpatError: unclosed token: line 1, column 0